Video display device and method of controlling the same

ABSTRACT

Provided are a video display device and a method of controlling the same. The method includes: obtaining an uncompressed audio-visual (AV) content; playing the uncompressed AV content through the display unit; obtaining content information of the uncompressed AV content on the basis of a section of the uncompressed AV content; obtaining an enhanced service on the basis of the content information; and playing the enhanced service through the display unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of U.S. application Ser.No. 13/883,239 (filed Jun. 21, 2013) which claims the benefit of U.S.Provisional Application No. 61/417,187 (filed on 2010 Nov. 24), U.S.Provisional Application No. 61/490,069 (filed on 2011 May 26), U.S.Provisional Application No. 61/525,836 (filed on 2011 Aug. 21), U.S.Provisional Application No. 61/525,837 (filed on 2011 Aug. 21), U.S.Provisional Application No. 61/549,238 (filed on 2011 Oct. 20), U.S.Provisional Application No. 61/532,535 (filed on 2011 Sep. 8), U.S.Provisional Application No. 61/536,066 (filed on 2011 Sep. 19), U.S.Provisional Application No. 61/549,237 (filed on 2011 Oct. 20), U.S.Provisional Application No. 61/417,192 (filed on 2010 Nov. 24), U.S.Provisional Application No. 61/525,838 (filed on 2011 Aug. 21) and PCTApplication No. PCT/KR2011/009045 (filed on 2011 Nov. 24) which arehereby incorporated by reference as if fully set forth herein.

BACKGROUND

The present disclosure relates to a video display device and a method ofcontrolling the same.

As digital broadcasting is paving the way for its extensive spread, abroadcasting station transmits both main audio-visual (AV) contents andenhanced service data, which may be linked to the main AV contents toprovide information and services or may be separately provided for otherpurposes.

However, a video display device in each home may unlikely receivebroadcast signals directly through air under a current broadcastingenvironment. Rather, most of cases, a video display device in each homeis connected to a broadcast receiving device to display uncompressedaudio-visual contents that the broadcast receiving device provides.

Additionally, the broadcast receiving device receives contents from aserver (called a Multichannel Video Programming Distributor (MVPD)). TheMVPD receives a broadcast signal from a broadcasting station, extractscontents from the received broadcast signal, converts the extractedcontent into signals having a proper format for transmission, andprovides the converted signals to a broadcast receiving device. Duringthese processes, the MVPD may exclude extracted enhanced service data ormay add another enhanced service data, so that the broadcasting stationmay serve to provide AV contents but may not provide enhanced servicesled by the broadcasting station.

Since the broadcast receiving device extracts main AV data from signalreceived from the MVPD and provides only uncompressed audio-visual datato a video display device, only enhanced services provided from thebroadcast receiving device not the video display device are available.Furthermore, the broadcasting station may not provide enhanced servicesled by itself.

Additionally, even when the broadcasting station provides enhancedservices through wireless channels, the enhanced services may not beprovided enough due to bandwidth limitation.

SUMMARY

Embodiments provide a video display device available for enhancedservices led by a broadcasting station and a method of controlling thesame.

In one embodiment, a method of controlling a video display deviceincluding a display unit includes: obtaining an uncompressedaudio-visual (AV) content; playing the uncompressed AV content throughthe display unit; obtaining content information of the uncompressed AVcontent on the basis of a section of the uncompressed AV content;obtaining an enhanced service on the basis of the content information;and playing the enhanced service through the display unit.

In further another embodiment, a video display device includes: adisplay unit; a receiving unit for obtaining an uncompressed AV content;an enhanced service management unit for obtaining content information ofthe uncompressed AV content on the basis of a section of theuncompressed AV content and obtaining an enhanced service on the basisof the content information; and a play controlling unit for playing theuncompressed AV content and the enhanced service on the display unit.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the network topology according tothe embodiment.

FIG. 2 is a block diagram illustrating a watermark based networktopology according to an embodiment.

FIG. 3 is a ladder diagram illustrating a data flow in a watermark basednetwork topology according to an embodiment.

FIG. 4 is view illustrating a watermark based content recognition timingaccording to an embodiment.

FIG. 5 is a block diagram illustrating a fingerprint based networktopology according to an embodiment.

FIG. 6 is a ladder diagram illustrating a data flow in a fingerprintbased network topology according to an embodiment.

FIG. 7 is a view illustrating an XML schema diagram of ACR-Resulttypecontaining a query result according to an embodiment.

FIG. 8 is a block diagram illustrating a watermark and fingerprint basednetwork topology according to an embodiment.

FIG. 9 is a ladder diagram illustrating a data flow in a watermark andfingerprint based network topology according to an embodiment.

FIG. 10 is a block diagram illustrating the video display deviceaccording to the embodiment.

FIG. 11 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

FIG. 12 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

FIG. 13 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

FIG. 14 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

FIG. 15 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

FIG. 16 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

FIG. 17 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

FIG. 18 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

FIG. 19 is a flowchart illustrating a method of synchronizing a playtime of a main AV content with a play time of an enhanced serviceaccording to an embodiment.

FIG. 20 is a conceptual diagram illustrating a method of synchronizing aplay time of a main AV content with a play time of an enhanced serviceaccording to an embodiment.

FIG. 21 is a screen when an auto playing application is executedaccording to an embodiment.

FIG. 22 is a screen when a content provider application is executedaccording to an embodiment.

FIG. 23 is a screen when a video display device manufacturer applicationhaving a resizing user interface is executed according to an embodiment.

FIG. 24 is a screen when a video display device manufacturer applicationhaving an overlay user interface is executed according to an embodiment.

FIG. 25 is a flowchart illustrating a method of executing a videodisplay device manufacturer application according to an embodiment.

FIG. 26 is a screen when an application selection menu is displayedaccording to an embodiment.

FIG. 27 is a screen when an all sub menu of a program main menu in avideo display device manufacturer application is selected according toan embodiment.

FIG. 28 is a screen when a participation sub menu of a program main menuin a video display device manufacturer application is selected accordingto an embodiment.

FIG. 29 is a screen when a discovery sub menu of a program main menu ina video display device manufacturer application is selected according toan embodiment.

FIG. 30 is flowchart illustrating a searching method according to anembodiment.

FIG. 31 is a screen when a search sub menu of an information main menuin a video display device manufacturer application is selected accordingto an embodiment.

FIG. 32 is another screen when a search sub menu of an information mainmenu in a video display device manufacturer application is selectedaccording to an embodiment.

FIG. 33 is flowchart illustrating a chatting window displaying methodaccording to an embodiment.

FIG. 34 is a screen when a Twitter sub menu of an information main menuin a video display device manufacturer application is selected accordingto an embodiment 6

FIG. 35 is a flowchart illustrating a news obtaining method according toan embodiment.

FIG. 36 is a screen when a news sub menu of an information main menu ina video display device manufacturer application is selected according toan embodiment.

FIG. 37 is flowchart illustrating a viewer list displaying methodaccording to an embodiment.

FIG. 38 is a screen when a viewer list sub menu of a social main menu ina video display device manufacturer application is selected according toan embodiment.

FIG. 39 is a signal flow when a video display device obtains a friendslist and watching information according to an embodiment.

FIG. 40 is a signal flow when a video display device obtains a friendslist and watching information according to another embodiment.

FIG. 41 is flowchart illustrating a preference displaying methodaccording to an embodiment.

FIG. 42 is a screen when a preference display sub menu of a social mainmenu in a video display device manufacturer application is selectedaccording to an embodiment.

FIG. 43 is flowchart illustrating a digital media downloading methodaccording to an embodiment.

FIG. 44 is a screen when a previous episode sub menu of a recommendationmain menu in a video display device manufacturer application is selectedaccording to an embodiment.

FIG. 45 is a screen when a VOD sub menu of a recommendation main menu ina video display device manufacturer application is selected according toan embodiment.

FIG. 46 is a screen when an application sub menu of a recommendationmain menu in a video display device manufacturer application is selectedaccording to an embodiment.

FIG. 47 is flowchart illustrating a catalog displaying method accordingto an embodiment.

FIG. 48 is a screen when a style sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

FIG. 49 is a screen when a music sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

FIG. 50 is a screen when another sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

FIG. 51 is a view of a user interface for controlling an ACR functionaccording to an embodiment.

FIG. 52 is a flowchart illustrating a method of controlling a videodisplay device according to the embodiment.

FIG. 53 is a flowchart illustrating an advertisement alternative methodaccording to an embodiment.

FIG. 54 is a flowchart illustrating an advertisement alternative methodaccording to another embodiment.

FIG. 55 is a flowchart illustrating a scene related informationdisplaying method according to an embodiment.

FIG. 56 is a user interface for selecting a scene related informationdisplaying mode according to an embodiment.

FIG. 57 is flowchart illustrating a scene related information displayingmethod according to an embodiment.

FIG. 58 is a flowchart illustrating a user selection displaying methodof scene related information according to an embodiment.

FIG. 59 is conceptual diagram illustrating a user selection displayingmethod of scene related information according to an embodiment.

FIG. 60 is a flowchart illustrating a method of displaying integratedscene related information according to an embodiment.

FIG. 61 is conceptual diagram illustrating a method of displayingintegrated scene related information according to an embodiment.

FIG. 62 illustrates an integrated screen of sponsor advertisements of amain AV content.

FIG. 63 is a flowchart illustrating a method of storing rating settinginformation according to an embodiment.

FIG. 64 is a user interface for storing rating setting informationaccording to an embodiment.

FIG. 65 is flowchart illustrating a method of playing a rating basedmain AV content according to an embodiment.

FIG. 66 is flowchart illustrating a method of playing a rating basedmain AV content according to another embodiment.

FIG. 67 is flowchart illustrating a method of playing a rating basedmain AV content according to another embodiment.

FIG. 68 is flowchart illustrating a method of registering to a usageinformation measuring server according to an embodiment.

FIG. 69 is a view illustrating an XML schema of user informationaccording to an embodiment.

FIG. 70 is flowchart illustrating a method of obtaining, reporting, andusing usage information according to an embodiment.

FIG. 71 is an XML schema of a main AV content watching information itemaccording to an embodiment.

FIG. 72 is an XML schema of an enhanced service usage information itemaccording to an embodiment.

FIGS. 73 to 76 are conceptual diagrams of a user characteristic basedenhanced service according to an embodiment.

FIG. 77 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

FIG. 78 is a block diagram illustrating a structure of a watermark basedvideo display device according to another embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, a mobile terminal relating to the present invention will bedescribed in more detail with reference to the accompanying drawings.Noun suffixes such as “engine”, “module”, and “unit” for components indescription below are given or mixed in consideration of easiness inwriting the specification. That is, the noun suffixes themselves doesnot have respectively distinguishable meanings or roles.

A network topology will be described with reference to FIGS. 1 to 9according to an embodiment.

FIG. 1 is a block diagram illustrating the network topology according tothe embodiment.

As shown in FIG. 1, the network topology includes a content providingserver 10, a content recognizing service providing server 20, a multichannel video distributing server 30, an enhanced service informationproviding server 40, a plurality of enhanced service providing servers50, a broadcast receiving device 60, a network 70, and a video displaydevice 100.

The content providing server 10 may correspond to a broadcasting stationand broadcasts a broadcast signal including main audio-visual contents.The broadcast signal may further include enhanced services. The enhancedservices may or may not relate to main audio-visual contents. Theenhanced services may have formats such as service information,metadata, additional data, compiled execution files, web applications,Hypertext Markup Language (HTML) documents, XML documents, CascadingStyle Sheet (CSS) documents, audio files, video files, ATSC 2.0contents, and addresses such as Uniform Resource Locator (URL). Theremay be at least one content providing server.

The content recognizing service providing server 20 provides a contentrecognizing service that allows the video display device 100 torecognize content on the basis of main audio-visual content. The contentrecognizing service providing server 20 may or may not edit the mainaudio-visual content. There may be at least one content recognizingservice providing server.

The content recognizing service providing server 20 may be a watermarkserver that edits the main audio-visual content to insert a visiblewatermark, which may look a logo, into the main audio-visual content.This watermark server may insert the logo of a content provider at theupper-left or upper-right of each frame in the main audio-visual contentas a watermark.

Additionally, the content recognizing service providing server 20 may bea watermark server that edits the main audio-visual content to insertcontent information into the main audio-visual content as an invisiblewatermark.

Additionally, the content recognizing service providing server 20 may bea fingerprint server that extracts feature information from some framesor audio samples of the main audio-visual content and stores it. Thisfeature information is called signature.

The multi channel video distributing server 30 receives and multiplexesbroadcast signals from a plurality of broadcasting stations and providesthe multiplexed broadcast signals to the broadcast receiving device 60.Especially, the multi channel video distributing server 30 performsdemodulation and channel decoding on the received broadcast signals toextract main audio-visual content and enhanced service, and then,performs channel encoding on the extracted main audio-visual content andenhanced service to generate a multiplexed signal for distribution. Atthis point, since the multi channel video distributing server 30 mayexclude the extracted enhanced service or may add another enhancedservice, a broadcasting station may not provide services led by it.There may be at least one multi channel video distributing server.

The broadcasting device 60 may tune a channel selected by a user andreceives a signal of the tuned channel, and then, performs demodulationand channel decoding on the received signal to extract a mainaudio-visual content. The broadcasting device 60 decodes the extractedmain audio-visual content through H.264/Moving Picture Experts Group-4advanced video coding (MPEG-4 AVC), Dolby AC-3 or Moving Picture ExpertsGroup-2 Advanced Audio Coding (MPEG-2 AAC) algorithm to generate anuncompressed main audio-visual (AV) content. The broadcast receivingdevice 60 provides the generated uncompressed main AV content to thevideo display device 100 through its external input port.

The enhanced service information providing server 40 provides enhancedservice information on at least one available enhanced service relatingto a main AV content in response to a request of a video display device.There may be at least one enhanced service providing server. Theenhanced service information providing server 40 may provide enhancedservice information on the enhanced service having the highest priorityamong a plurality of available enhanced services.

The enhanced service providing server 50 provides at least one availableenhanced service relating to a main AV content in response to a requestof a video display device. There may be at least one enhanced serviceproviding server.

The video display device 100 may be a television, a notebook computer, ahand phone, and a smart phone, each including a display unit. The videodisplay device 100 may receive an uncompressed main AV content from thebroadcast receiving device 60 or a broadcast signal including an encodedmain AV content from the contents providing server 10 or the multichannel video distributing server 30. The video display device 100 mayreceive a content recognizing service from the content recognizingservice providing server 20 through the network 70, an address of atleast one available enhanced service relating to a main AV content fromthe enhanced service information providing server 40 through the network70, and at least one available enhanced service relating to a main AVcontent from the enhanced service providing server 50.

At least two of the content providing server 10, the content recognizingservice providing server 20, the multi channel video distributing server30, the enhanced service information providing server 40, and theplurality of enhanced service providing servers 50 may be combined in aform of one server and may be operated by one provider.

FIG. 2 is a block diagram illustrating a watermark based networktopology according to an embodiment.

As shown in FIG. 2, the watermark based network topology may furtherinclude a watermark server 21.

As shown in FIG. 2, the watermark server 21 edits a main AV content toinsert content information into it. The multi channel video distributingserver 30 may receive and distribute a broadcast signal including themodified main AV content. Especially, a watermark server may use adigital watermarking technique described below.

A digital watermark is a process for inserting information, which may bealmost undeletable, into a digital signal. For example, the digitalsignal may be audio, picture, or video. If the digital signal is copied,the inserted information is included in the copy. One digital signal maycarry several different watermarks simultaneously.

In visible watermarking, the inserted information may be identifiable ina picture or video. Typically, the inserted information may be a text orlogo identifying a media owner. If a television broadcasting stationadds its logo in a corner of a video, this is an identifiable watermark.

In invisible watermarking, although information as digital data is addedto audio, picture, or video, a user may be aware of a predeterminedamount of information but may not recognize it. A secret message may bedelivered through the invisible watermarking.

One application of the watermarking is a copyright protection system forpreventing the illegal copy of digital media. For example, a copy deviceobtains a watermark from digital media before copying the digital mediaand determines whether to copy or not on the bases of the content of thewatermark.

Another application of the watermarking is source tracking of digitalmedia. A watermark is embedded in the digital media at each point of adistribution path. If such digital media is found later, a watermark maybe extracted from the digital media and a distribution source may berecognized from the content of the watermark.

Another application of invisible watermarking is a description fordigital media.

A file format for digital media may include additional informationcalled metadata and a digital watermark is distinguished from metadatain that it is delivered as an AV signal itself of digital media.

The watermarking method may include spread spectrum, quantization, andamplitude modulation.

If a marked signal is obtained through additional editing, thewatermarking method corresponds to the spread spectrum. Although it isknown that the spread spectrum watermark is quite strong, not muchinformation is contained because the watermark interferes with anembedded host signal.

If a marked signal is obtained through the quantization, thewatermarking method corresponds to a quantization type. The quantizationwatermark is weak, much information may be contained.

If a marked signal is obtained through an additional editing methodsimilar to the spread spectrum in a spatial domain, a watermarkingmethod corresponds to the amplitude modulation.

FIG. 3 is a ladder diagram illustrating a data flow in a watermark basednetwork topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main AV content and an enhanced service in operation S101.

The watermark server 21 receives a broadcast signal that the contentproviding server 10 provides, inserts a visible watermark such as a logoor watermark information as an invisible watermark into the main AVcontent by editing the main AV content, and provides the watermarkedmain AV content and enhanced service to the MVPD 30 in operation S103.

The watermark information inserted through an invisible watermark mayinclude at least one of a watermark purpose, content information,enhanced service information, and an available enhanced service. Thewatermark purpose represents one of illegal copy prevention, viewerratings, and enhanced service acquisition.

The content information may include at least one of identificationinformation of a content provider that provides main AV content, main AVcontent identification information, time information of a contentsection used in content information acquisition, names of channelsthrough which main AV content is broadcasted, logos of channels throughwhich main AV content is broadcasted, descriptions of channels throughwhich main AV content is broadcasted, a usage information reportingperiod, the minimum usage time for usage information acquisition, andavailable enhanced service information relating to main AV content.

If the video display device 100 uses a watermark to acquire contentinformation, the time information of a content section used for contentinformation acquisition may be the time information of a content sectioninto which a watermark used is embedded. If the video display device 100uses a fingerprint to acquire content information, the time informationof a content section used for content information acquisition may be thetime information of a content section where feature information isextracted. The time information of a content section used for contentinformation acquisition may include at least one of the start time of acontent section used for content information acquisition, the durationof a content section used for content information acquisition, and theend time of a content section used for content information acquisition.

The usage information reporting address may include at least one of amain AV content watching information reporting address and an enhancedservice usage information reporting address. The usage informationreporting period may include at least one of a main AV content watchinginformation reporting period and an enhanced service usage informationreporting period. A minimum usage time for usage information acquisitionmay include at least one of a minimum watching time for a main AVcontent watching information acquisition and a minimum usage time forenhanced service usage information extraction.

On the basis that a main AV content is watched for more than the minimumwatching time, the video display device 100 acquires watchinginformation of the main AV content and reports the acquired watchinginformation to the main AV content watching information reportingaddress in the main AV content watching information reporting period.

On the basis that an enhanced service is used for more than the minimumusage time, the video display device 100 acquires enhanced service usageinformation and reports the acquired usage information to the enhancedservice usage information reporting address in the enhanced serviceusage information reporting period.

The enhanced service information may include at least one of informationon whether an enhanced service exists, an enhanced service addressproviding server address, an acquisition path of each available enhancedservice, an address for each available enhanced service, a start time ofeach available enhanced service, an end time of each available enhancedservice, a lifetime of each available enhanced service, an acquisitionmode of each available enhanced service, a request period of eachavailable enhanced service, priority information each available enhancedservice, description of each available enhanced service, a category ofeach available enhanced service, a usage information reporting address,a usage information reporting period, and the minimum usage time forusage information acquisition.

The acquisition path of available enhanced service may be representedwith IP or Advanced Television Systems Committee-Mobile/Handheld (ATSCM/H). If the acquisition path of available enhanced service is ATSC M/H,enhanced service information may further include frequency informationand channel information. An acquisition mode of each available enhancedservice may represent Push or Pull.

Moreover, the watermark server 21 may insert watermark information as aninvisible watermark into the logo of a main AV content.

For example, the watermark server 21 may insert a barcode at apredetermined position of a logo. At this point, the predeterminedposition of the logo may correspond to the first line at the bottom ofan area where the logo is displayed. The video display device 100 maynot display a barcode when receiving a main AV content including a logowith the barcode inserted.

For example, the watermark server 21 may insert a barcode at apredetermined position of a logo. At this point, the log may maintainits form.

For example, the watermark server 21 may insert N-bit watermarkinformation at each of the logos of M frames. That is, the watermarkserver 21 may insert M*N watermark information in M frames.

The MVPD 30 receives broadcast signals including watermarked main AVcontent and enhanced service and generates a multiplexed signal toprovide it to the broadcast receiving device 60 in operation S105. Atthis point, the multiplexed signal may exclude the received enhancedservice or may include new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main AV content, and then, providesthe generated uncompressed main AV content to the video display device100 in operation S106.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main AV content through a wireless channel inoperation S107.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main AV content to the video display device 100 withoutgoing through the broadcast receiving device 60 in operation S108.

The video display device 100 may receive an uncompressed main AV contentthrough the broadcast receiving device 60. Additionally, the videodisplay device 100 may receive a broadcast signal through a wirelesschannel, and then, may demodulate and decode the received broadcastsignal to obtain a main AV content. Additionally, the video displaydevice 100 may receive a broadcast signal from the MVPD 30, and then,may demodulate and decode the received broadcast signal to obtain a mainAV content. The video display device 100 extracts watermark informationfrom some frames or a section of audio samples of the obtained main AVcontent. If watermark information corresponds to a logo, the videodisplay device 100 confirms a watermark server address corresponding toa logo extracted from a corresponding relationship between a pluralityof logos and a plurality of watermark server addresses. When thewatermark information corresponds to the logo, the video display device100 cannot identify the main AV content only with the logo.Additionally, when the watermark information does not include contentinformation, the video display device 100 cannot identify the main AVcontent but the watermark information may include content provideridentifying information or a watermark server address. When thewatermark information includes the content provider identifyinginformation, the video display device 100 may confirm a watermark serveraddress corresponding to the content provider identifying informationextracted from a corresponding relationship between a plurality ofcontent provider identifying information and a plurality of watermarkserver addresses. In this manner, when the video display device 100cannot identify a main AV content the video display device 100 only withthe watermark information, it accesses the watermark server 21corresponding to the obtained watermark server address to transmit afirst query in operation S109.

The watermark server 21 provides a first reply to the first query inoperation S111. The first reply may include at least one of contentinformation, enhanced service information, and an available enhancedservice.

If the watermark information and the first reply do not include anenhanced service address, the video display device 100 cannot obtainenhanced service. However, the watermark information and the first replymay include an enhanced service address providing server address. Inthis manner, the video display device 100 does not obtain a serviceaddress or enhanced service through the watermark information and thefirst reply. If the video display device 100 obtains an enhanced serviceaddress providing server address, it accesses the enhanced serviceinformation providing server 40 corresponding to the obtained enhancedservice address providing server address to transmit a second queryincluding content information in operation S119.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe second query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a secondreply to the second query in operation S121.

If the video display device 100 obtains at least one available enhancedservice address through the watermark information, the first reply, orthe second reply, it accesses the at least one available enhancedservice address to request enhanced service in operation S123, and then,obtains the enhanced service in operation S125.

FIG. 4 is a view illustrating a watermark based content recognitiontiming according to an embodiment.

As shown in FIG. 4, when the broadcast receiving device 60 is turned onand tunes a channel, and also, the video display device 100 receives amain AV content of the turned channel from the broadcast receivingdevice 60 through an external input port 111, the video display device100 may sense a content provider identifier (or a broadcasting stationidentifier) from the watermark of the main AV content. Then, the videodisplay device 100 may sense content information from the watermark ofthe main AV content on the basis of the sensed content provideridentifier.

At this point, as shown in FIG. 4, the detection available period of thecontent provider identifier may be different from that of the contentinformation. Especially, the detection available period of the contentprovider identifier may be shorter than that of the content information.Through this, the video display device 100 may have an efficientconfiguration for detecting only necessary information.

FIG. 5 is a block diagram illustrating a fingerprint based networktopology according to an embodiment.

As shown in FIG. 5, the network topology may further include afingerprint server 22.

As shown in FIG. 5, the fingerprint server 22 does not edit a main AVcontent, but extracts feature information from some frames or a sectionof audio samples of the main AV content and stores the extracted featureinformation. Then, when receiving the feature information from the videodisplay device 100, the fingerprint server 22 provides an identifier andtime information of an AV content corresponding to the received featureinformation.

FIG. 6 is a ladder diagram illustrating a data flow in a fingerprintbased network topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main AV content and an enhanced service in operation S201.

The fingerprint server 22 receives a broadcast signal that the contentproviding server 10, extracts a plurality of pieces of featureinformation from a plurality of frame sections or a plurality of audiosections of the main AV content, and establishes a database for aplurality of query results corresponding to the plurality of featureinformation in operation S203. The query result may include at least oneof content information, enhanced service information, and an availableenhanced service.

The MVPD 30 receives broadcast signals including a main AV content andenhanced service and generates a multiplexed signal to provide it to thebroadcast receiving device 60 in operation S205. At this point, themultiplexed signal may exclude the received enhanced service or mayinclude new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main AV content, and then, providesthe generated uncompressed main AV content to the video display device100 in operation S206.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main AV content through a wireless channel inoperation S207.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main AV content to the video display device 100 withoutgoing through the broadcast receiving device 60.

The video display device 100 may receive an uncompressed main AV contentthrough the broadcast receiving device 60. Additionally, the videodisplay device 100 may receive a broadcast signal through a wirelesschannel, and then, may demodulate and decode the received broadcastsignal to obtain a main AV content. Additionally, the video displaydevice 100 may receive a broadcast signal from the MVPD 30, and then,may demodulate and decode the received broadcast signal to obtain a mainAV content. The video display device 100 extracts feature informationfrom some frames or a section of audio samples of the obtained main AVcontent in operation S213.

The video display device 100 accesses the fingerprint server 22corresponding to the predetermined fingerprint server address totransmit a first query including the extracted feature information inoperation S215.

The fingerprint server 22 provides a query result as a first reply tothe first query in operation S217. If the first reply corresponds tofail, the video display device 100 accesses the fingerprint server 22corresponding to another fingerprint server address to transmit a firstquery including the extracted feature information.

The fingerprint server 22 may provide Extensible Markup Language (XML)document as a query result. Examples of the XML document containing aquery result will be described with reference to FIG. 7 and Table 1.

FIG. 7 is a view illustrating an XML schema diagram of ACR-Resulttypecontaining a query result according to an embodiment.

As shown in FIG. 7, ACR-Resulttype containing a query result includesResultCode attributes and ContentID, NTPTimestamp,SignalingChannelInformation, and ServiceInformation elements.

For example, if the ResultCode attribute has 200, this may mean that thequery result is successful. For example, if the ResultCode attribute has404, this may mean that the query result is unsuccessful.

The SignalingChannelInformation element includes a SignalingChannelURL,and the SignalingChannelURL element includes an UpdateMode andPollingCycle attributes. The UpdateMode attribute may have a Pull valueor a Push value.

The ServiceInformation element includes ServiceName, ServiceLogo, andServiceDescription elements.

Table 1 illustrates an XML schema of ACR-ResultType containing the queryresult.

TABLE 1   <xs:complexType name=“ACR-ResultType”>   <xs:sequence>   <xs:element name=“ContentID” type=“xs:anyURI”/>    <xs:elementname=“NTPTimestamp” type=“xs:unsignedLong”/>    <xs:elementname=“SignalingChannelInformation”>     <xs:complexType>     <xs:sequence>       <xs:element name=“SignalingChannelURL”maxOccurs=“unbounded”>          <xs:complexType>        <xs:simpleContent>          <xs:extension base=“xs:anyURI”>          <xs:attribute name=“UpdateMode”>            <xs:simpleType>            <xs:restriction base=“xs:string”>             <xs:enumeration value=“Pull”/>              <xs:enumerationvalue=“Push”/>             </xs:restriction>            </xs:simpleType>          </xs:attribute>           <xs:attribute name=“PollingCycle”type=“xs:unsignedInt”/>          </xs:extension>        </xs:simpleContent>        </xs:complexType>       </xs:element>     </xs:sequence>     </xs:complexType>    </xs:element>   <xs:element name=“ServiceInformation”>     <xs:complexType>     <xs:sequence>       <xs:element name=“ServiceName”type=“xs:string”/>       <xs:element name=“ServiceLogo” type=“xs:anyURI”minOccurs=“0”/>       <xs:element name=“ServiceDescription”type=“xs:string” minOccurs=“0” maxOccurs=“unbounded”/>     </xs:sequence>     </xs:complexType>    </xs:element>    <xs:anynamespace=“##other” processContents=“skip” minOccurs=“0”maxOccurs=“unbounded”/>   </xs:sequence>   <xs:attributename=“ResultCode” type=“xs:string” use=“required”/>   <xs:anyAttributeprocessContents=“skip”/>  </xs:complexType>

As the ContentID element, an ATSC content identifier may be used asshown in Table 2.

TABLE 2 Syntax The Number of bits format ATSC_content_identifier( ) { TSID 16  uimsbf  reserved 2 bslbf  end_of_day 5 uimsbf  unique_for 9uimsbf  content_id var }

As shown in Table 2, the ATSC content identifier has a structureincluding TSID and a house number.

The 16 bit unsigned integer TSID carries a transport stream identifier.

The 5 bit unsigned integer end_of_day is set with an hour in a day ofwhen a content_id value can be reused after broadcasting is finished.

The 9 bit unsigned integer unique_for is set with the number of day ofwhen the content_id value cannot be reused.

Content_id represents a content identifier. The video display device 100reduces unique_for by 1 in a corresponding time to end_of_day daily andpresumes that content_id is unique if unique_for is not 0.

Moreover, as the ContentID element, a global service identifier forATSC-M/H service may be used as described below.

The global service identifier has the following form.

-   -   urn:oma:bcast:iauth:atsc:service:<region>:<xsid>:<serviceid>

Here, <region> is an international country code including two charactersregulated by ISO 639-2. <xsid> for local service is a decimal number ofTSID as defined in <region>, and <xsid> (regional service) (major>69) is“0”. <serviceid> is defined with <major> or <minor>. <major> represent aMajor Channel number, and <minor> represents a Minor Channel Number.

Examples of the global service identifier are as follows.

-   -   urn:oma:bcast:iauth:atsc:service:us:1234:5.1    -   urn:oma:bcast:iauth:atsc:service:us:0:100.200

Moreover, as the ContentID element, an ATSC content identifier may beused as described below.

The ATSC content identifier has the following form.

urn:oma:bcast:iauth:atsc:content:<region>:<xsidz>:<contentid>:<unique_for>:<end_of_day>

Here, <region> is an international country code including two charactersregulated by ISO 639-2. <xsid> for local service is a decimal number ofTSID as defined in <region>, and may be followed by “.”<serviceid>.<xsid> for (regional service) (major>69) is <serviceid>. <content_id> isa base64 sign of a content_id field defined in Table 2, <unique_for> isa decimal number sign of an unique_for field defined in Table 2, and<end_of_day> is a decimal number sign of an end_of_day field defined inTable 2.

Hereinafter, FIG. 6 is described again.

If the query result does not include an enhanced service address orenhanced service but includes an enhanced service address providingserver address, the video display device 100 accesses the enhancedservice information providing server 40 corresponding to the obtainedenhanced service address providing server address to transmit a secondquery including content information in operation S219.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe second query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a secondreply to the second query in operation S221.

If the video display device 100 obtains at least one available enhancedservice address through the first reply or the second reply, it accessesthe at least one available enhanced service address to request enhancedservice in operation S223, and then, obtains the enhanced service inoperation S225.

When the UpdateMode attribute has a Pull value, the video display device100 transmits an HTTP request to the enhanced service providing server50 through SignalingChannelURL and receives an HTTP reply including aPSIP binary stream from the enhanced service providing server 50 inresponse to the request. In this case, the video display device 100 maytransmit the HTTP request according to a Polling period designated asthe PollingCycle attribute. Additionally, the SignalingChannelURLelement may have an update time attribute. In this case, the videodisplay device 100 may transmit the HTTP request according to an updatetime designated as the update time attribute.

If the UpdateMode attribute has a Push value, the video display device100 may receive update from a server asynchronously throughXMLHTTPRequest API. After the video display device 100 transmits anasynchronous request to a server through XMLHTTPRequest object, if thereis a change of signaling information, the server provides the signalinginformation as a reply through the channel. If there is limitation insession standby time, a server generates a session timeout reply and areceiver recognizes the generated timeout reply to transmit a requestagain, so that a signaling channel between the receiver and the servermay be maintained for all time.

FIG. 8 is a block diagram illustrating a watermark and fingerprint basednetwork topology according to an embodiment.

As shown in FIG. 8, the watermark and fingerprint based network topologymay further include a watermark server 21 and a fingerprint server 22.

As shown in FIG. 8, the watermark server 21 inserts content provideridentifying information into a main AV content. The watermark server 21may insert content provider identifying information as a visiblewatermark such as a logo or an invisible watermark into a main AVcontent.

The fingerprint server 22 does not edit a main AV content, but extractsfeature information from some frames or a certain section of audiosamples of the main AV content and stores the extracted featureinformation. Then, when receiving the feature information from the videodisplay device 100, the fingerprint server 22 provides an identifier andtime information of an AV content corresponding to the received featureinformation.

FIG. 9 is a ladder diagram illustrating a data flow in a watermark andfingerprint based network topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main AV content and an enhanced service in operation S301.

The watermark server 21 receives a broadcast signal that the contentproviding server 10 provides, inserts a visible watermark such as a logoor watermark information as an invisible watermark into the main AVcontent by editing the main AV content, and provides the watermarkedmain AV content and enhanced service to the MVPD 30 in operation S303.The watermark information inserted through an invisible watermark mayinclude at least one of content information, enhanced serviceinformation, and an available enhanced service. The content informationand enhanced service information are described above.

The MVPD 30 receives broadcast signals including watermarked main AVcontent and enhanced service and generates a multiplexed signal toprovide it to the broadcast receiving device 60 in operation S305. Atthis point, the multiplexed signal may exclude the received enhancedservice or may include new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main AV content, and then, providesthe generated uncompressed main AV content to the video display device100 in operation S306.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main AV content through a wireless channel inoperation S307.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main AV content to the video display device 100 withoutgoing through the broadcast receiving device 60 in operation S308.

The video display device 100 may receive an uncompressed main AV contentthrough the broadcast receiving device 60. Additionally, the videodisplay device 100 may receive a broadcast signal through a wirelesschannel, and then, may demodulate and decode the received broadcastsignal to obtain a main AV content. Additionally, the video displaydevice 100 may receive a broadcast signal from the MVPD 30, and then,may demodulate and decode the received broadcast signal to obtain a mainAV content. The video display device 100 extracts watermark informationfrom audio samples in some frames or periods of the obtained main AVcontent. If watermark information corresponds to a logo, the videodisplay device 100 confirms a watermark server address corresponding toa logo extracted from a corresponding relationship between a pluralityof logos and a plurality of watermark server addresses. When thewatermark information corresponds to the logo, the video display device100 cannot identify the main AV content only with the logo.Additionally, when the watermark information does not include contentinformation, the video display device 100 cannot identify the main AVcontent but the watermark information may include content provideridentifying information or a watermark server address. When thewatermark information includes the content provider identifyinginformation, the video display device 100 may confirm a watermark serveraddress corresponding to the content provider identifying informationextracted from a corresponding relationship between a plurality ofcontent provider identifying information and a plurality of watermarkserver addresses. In this manner, when the video display device 100cannot identify a main AV content the video display device 100 only withthe watermark information, it accesses the watermark server 21corresponding to the obtained watermark server address to transmit afirst query in operation S309.

The watermark server 21 provides a first reply to the first query inoperation S311. The first reply may include at least one of afingerprint server address, content information, enhanced serviceinformation, and an available enhanced service. The content informationand enhanced service information are described above.

If the watermark information and the first reply include a fingerprintserver address, the video display device 100 extracts featureinformation from some frames or a certain section of audio samples ofthe main AV content in operation S313.

The video display device 100 accesses the fingerprint server 22corresponding to the fingerprint server address in the first reply totransmit a second query including the extracted feature information inoperation S315.

The fingerprint server 22 provides a query result as a second reply tothe second query in operation S317.

If the query result does not include an enhanced service address orenhanced service but includes an enhanced service address providingserver address, the video display device 100 accesses the enhancedservice information providing server 40 corresponding to the obtainedenhanced service address providing server address to transmit a thirdquery including content information in operation S319.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe third query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a third replyto the third query in operation S321.

If the video display device 100 obtains at least one available enhancedservice address through the first reply, the second reply, or the thirdreply, it accesses the at least one available enhanced service addressto request enhanced service in operation S323, and then, obtains theenhanced service in operation S325.

Then, referring to FIGS. 10 and 18, the video display device 100 will bedescribed according to an embodiment.

FIG. 10 is a block diagram illustrating the video display deviceaccording to the embodiment.

As shown in FIG. 10, the video display device 100 includes a broadcastsignal receiving unit 101, a demodulation unit 103, a channel decodingunit 105, a demultiplexing unit 107, an AV decoding unit 109, anexternal input port 111, a play controlling unit 113, a play device 120,an enhanced service management unit 130, a data transmitting/receivingunit 141, and a memory 150.

The broadcast signal receiving unit 101 receives a broadcast signal fromthe content providing server 10 or MVPD 30.

The demodulation unit 103 demodulates the received broadcast signal togenerate a demodulated signal.

The channel decoding unit 105 performs channel decoding on thedemodulated signal to generate channel-decoded data.

The demultiplexing unit 107 separates a main AV content and enhancedservice from the channel-decoded data. The separated enhanced service isstored in an enhanced service storage unit 152.

The AV decoding unit 109 performs AV decoding on the separated main AVcontent to generate an uncompressed main AV content.

Moreover, the external input port 111 receives an uncompressed main AVcontent from the broadcast receiving device 60, a digital versatile disk(DVD) player, a Blu-ray disk player, and so on. The external input port111 may include at least one of a DSUB port, a High DefinitionMultimedia Interface (HDMI) port, a Digital Visual Interface (DVI) port,a composite port, a component port, and an S-Video port.

The play controlling unit 113 controls the play device 120 to play atleast one of an uncompressed main AV content that the AV decoding unit109 generates and an uncompressed main AV content received from theexternal input port 111 according to a user's selection.

The play device 120 includes a display unit 121 and a speaker 123. Thedisplay unit 21 may include at least one of a liquid crystal display(LCD), a thin film transistor-liquid crystal display (TFT LCD), anorganic light-emitting diode (OLED), a flexible display, and a 3Ddisplay.

The enhanced service management unit 130 obtains content information ofthe main AV content and obtains available enhanced service on the basisof the obtained content information. Especially, as described above, theenhanced service management unit 130 may obtain the identificationinformation of the main AV content on the basis of some frames or acertain section of audio samples the uncompressed main AV content. Thisis called automatic contents recognition (ACR) in this specification.

The data transmitting/receiving unit 141 may include an AdvancedTelevision Systems Committee-Mobile/Handheld (ATSC-M/H) channeltransmitting/receiving unit 141 a and an IP transmitting/receiving unit141 b.

The memory 150 may include at least one type of storage medium such as aflash memory type, a hard disk type, a multimedia card micro type, acard type memory such as SD or XD memory, Random Access Memory (RAM),Static Random Access Memory (SRAM), Read-Only Memory (ROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM), Programmable Read-OnlyMemory (PROM), magnetic memory, magnetic disk, and optical disk. Thevideo display device 100 may operate in linkage with a web storageperforming a storage function of the memory 150 in the Internet.

The memory 150 may include a content information storage unit 151, anenhanced service storage unit 152, a logo storage unit 153, a settinginformation storage unit 154, a bookmark storage unit 155, a userinformation storage unit 156, and a usage information storage unit 157.

The content information storage unit 151 stores a plurality of contentinformation corresponding to a plurality of feature information.

The enhanced service storage unit 152 may store a plurality of enhancedservices corresponding to a plurality of feature information or aplurality of enhanced services corresponding to a plurality of contentinformation.

The logo storage unit 153 stores a plurality of logos. Additionally, thelogo storage unit 153 may further store content provider identifierscorresponding to the plurality of logos or watermark server addressescorresponding to the plurality of logos.

The setting information storage unit 154 stores setting information forACR.

The bookmark storage unit 155 stores a plurality of bookmarks.

The user information storage unit 156 stores user information. The userinformation may include at least one of at least one account informationfor at least one service, regional information, family memberinformation, preferred genre information, video display deviceinformation, and a usage information range. The at least one accountinformation may include account information for a usage informationmeasuring server and account information of social network service suchas Twitter and Facebook. The regional information may include addressinformation and zip codes. The family member information may include thenumber of family members, each member's age, each member's sex, eachmember's religion, and each member's job. The preferred genreinformation may be set with at least one of sports, movie, drama,education, news, entertainment, and other genres. The video displaydevice information may include information such as the type,manufacturer, firmware version, resolution, model, OS, browser, storagedevice availability, storage device capacity, and network speed of avideo display device. Once the usage information range is set, the videodisplay device 100 collects and reports main AV content watchinginformation and enhanced service usage information within the set range.The usage information range may be set in each virtual channel.Additionally, the usage information measurement allowable range may beset over an entire physical channel.

The usage information providing unit 157 stores the main AV contentwatching information and the enhanced service usage information, whichare collected by the video display device 100. Additionally, the videodisplay device 100 analyzes a service usage pattern on the basis of thecollected main AV content watching information and enhanced serviceusage information, and stores the analyzed service usage pattern in theusage information storage unit 157.

The enhanced service management unit 130 may obtain the contentinformation of the main AV content from the fingerprint server 22 or thecontent information storage unit 151. If there is no content informationor sufficient content information, which corresponds to the extractedfeature information, in the content information storage unit 151, theenhanced service management unit 130 may receive additional contentinformation through the data transmitting/receiving unit 141. Moreover,the enhanced service management unit 130 may update the contentinformation continuously.

The enhanced service management unit 130 may obtain available enhancedservice from the enhanced service providing server 50 or the enhancedservice storage unit 153. If there is no enhanced service or sufficientenhanced service in the enhanced service storage unit 153, the enhancedservice management unit 130 may update enhanced service through the datatransmitting/receiving unit 141. Moreover, the enhanced servicemanagement unit 130 may update the enhanced service continuously.

The enhanced service management unit 130 may extracts a logo from themain AV content, and then, may make a query to the logo storage unit 155to obtain a content provider identifier or watermark server address,which is corresponds to the extracted logo. If there is no logo or asufficient logo, which corresponds to the extracted logo, in the logostorage unit 155, the enhanced service management unit 130 may receivean additional logo through the data transmitting/receiving unit 141.Moreover, the enhanced service management unit 130 may update the logocontinuously.

The enhanced service management unit 130 may compare the logo extractedfrom the main AV content with the plurality of logos in the logo storageunit 155 through various methods. The various methods may reduce theload of the comparison operation.

For example, the enhanced service management unit 130 may perform thecomparison on the basis of color characteristics. That is, the enhancedservice management unit 130 may compare the color characteristic of theextracted logo with the color characteristics of the logos in the logostorage unit 155 to determine whether they are identical or not.

Moreover, the enhanced service management unit 130 may perform thecomparison on the basis of character recognition. That is, the enhancedservice management unit 130 may compare the character recognized fromthe extracted logo with the characters recognized from the logos in thelogo storage unit 155 to determine whether they are identical or not.

Furthermore, the enhanced service management unit 130 may perform thecomparison on the basis of the contour of the logo. That is, theenhanced service management unit 130 may compare the contour of theextracted logo with the contours of the logos in the logo storage unit155 to determine whether they are identical or not.

Hereinafter, referring to FIGS. 11 and 18, the enhanced servicemanagement unit 130 will be described according to various embodiments.

FIG. 11 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

As shown in FIG. 11, the enhanced service management unit 130 mayinclude an operating system 131 a, an audio driver 132 a, a video driver133 a, a network driver 134 a, an ACR platform application programinterface (API) 135 a, at least one ACR engine 136 a, an ACR daemon 137a, an ACR middleware 138 a, and at least one ACR application 139 a.

When the enhanced service management unit 130 is used as shown in FIG.11, a user doesn't need to necessarily recognize a channel or contentand an ACR operation may be performed always.

The enhanced service management unit 130 may access an application storethrough the data transmitting/receiving unit 141 to downloadapplication.

The audio driver 132 a may access an audio buffer used for audiofingerprint or audio watermark.

The video driver 133 a may access a video buffer used for videofingerprint or video watermark.

The network driver 134 a allows the ACR engine 136 a to access the datatransmitting/receiving unit 141.

The platform API 135 a provides API to allow the ACR engine 136 a toaccess the audio driver 132 a, the video driver 133 a, and the networkdriver 134 a.

Since a plurality of broadcasting stations or content providers userespectively different content recognizing services, the video displaydevice 100 may include a plurality of ACR engines 136 a. That is, thevideo display device 100 may include an ACR engine for at least one of aplurality of video watermark extraction algorithms, a plurality of audiowatermark extraction algorithms, a plurality of video signatureextraction algorithms, and a plurality of audio signature extractionalgorithms. The watermark based ACR engine 136 a may extract watermarkinformation, and then, may obtain at least one of a watermark purpose,content information, enhanced service information, and an availableenhanced service from the extracted watermark information. Thefingerprint based ACR engine 136 a may extract feature information of acertain section of a main AV content, and then, may obtain at least oneof content information, enhanced service information, and an availableenhanced service on the basis of the extracted feature information.

The ACR daemon 137 a manages at least one ACR engine 136 a. The daemonis executed continuously to process a periodic service request. Thedaemon appropriately delivers the collected requests to allow anotherprogram to process them. The ACR daemon 137 a may manage the executionand termination of the ACR engine 136 a. The ACR daemon 137 a searchesan ACR engine matching to a current main AV content among a plurality ofACR engines 136 a. Especially, if a system resource is consumed greatlybecause several ACR engines are executed, the ACR daemon 137 asequentially executes the plurality of ACR engines 136 a one by oneaccording to a specific rule or priority to confirm whether contentrecognition is successful. Additionally, if one of the plurality of ACRengines 136 a is successful in content recognition, the ACR daemon 137 aobtains and executes an ACR application 139 a corresponding to anenhanced service for the recognized content. While the ACR engine 136 aperforms the content recognition successfully, if contents are notrecognized any more due to a channel change, the ACR daemon 137 a mayterminate the ACR application 139 a.

The ACR middleware 138 a serves as a browser for at least one ACRapplication 139 a.

The ACR application 139 a may provide user experience enhanced by ACR toa user. The ACR application 139 a may be an application downloaded froman application store or the Internet or may be a simple Uniform ResourceLocator (URL). The ACR application 139 a may provide content targetadvertisement, a content provider portal service, a program specificservice such as electronic program guide (EPG), or a content providerspecific service. While the ACR application 139 a interact with the ACRengine 136 a through the ACR application, it receives contentinformation or signature from the ACR engine 136 a and additionallyobtains an enhanced service corresponding to the content information orsignature to play them in the play device 120. For this, an ACRapplication interface that the ACR application 139 a uses is shown inTable 3 below.

TABLE 3 Name description Method String Return recognized contentgetContentID( ) id, or undefined if content has not been recognized.Date getTiming( ) Return time stamp, or undefined if content has notbeen recognized or time stamp is unavailable. String Return thesignature getSignature( ) captured, or undefined if capturing of thesignature has failed.

FIG. 12 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

When the broadcast receiving device 60 and the video display device 100are turned on and the external input port 111 of the video displaydevice 100 is selected, the ACR daemon 137 a obtains the contentinformation of a main AV content broadcasted in a tuned channel throughat least one ACR engine 136 a in operation S401.

Then, the ACR daemon 137 a obtains an available enhanced servicerelating to the content information in operation S403.

The ACR daemon 137 a executes the obtained enhanced service inbackground in operation S405.

The ACR daemon 137 a displays a color button on the display unit 121 tonotify a user that enhanced service is available in operation S407.

When the ACR daemon 137 a receives the use of the enhanced service, itplays the enhanced service executed in background in the play device 120in operation S409.

If a change or main AV content is changed, the ACR daemon 137 arecognizes a change through at least one ACR engine 136 a and terminatesthe enhanced service in operation S411.

FIG. 13 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

As shown in FIG. 13, the enhanced service management unit 130 mayinclude an operating system 131 b, an audio driver 132 b, a video driver133 b, a network driver 134 b, an ACR platform application programinterface (API) 135 b, at least one ACR engine 136 b, an ACR enginemanagement unit 137 b, an ACR middleware 138 b, and at least one ACRapplication 139 b.

When the enhanced service management unit 130 is used as shown in FIG.13, a user may need to necessarily recognize a channel or content.However, since the number of queries for ACR may be reduced, serveroverhead may be decreased.

The audio driver 132 b, the video driver 133 b, the network driver 134b, the ACR platform API 135 b, the at least one ACR engine 136 b, andthe ACR middleware 138 b in the enhanced service management unit 130 ofFIG. 13 are identical or similar to those 132 a, 133 a, 134 a, 135 a,136 a, and 138 a in the enhanced service management unit 130 FIG. 11.Therefore, their detailed descriptions will be omitted.

The ACR engine management unit 137 b manages a list of at least one ACRengine 136 b and provides a selection interface of the ACR engine 136 bto the ACR application 139 b.

The ACR application 139 b may provide user experience enhanced by ACR toa user. The ACR application 139 b may be an application downloaded froman application store or the Internet or may be a simple Uniform ResourceLocator (URL). The ACR application 139 b may provide content targetadvertisement, a content provider portal service, a program specificservice such as electronic program guide (EPG), or a content providerspecific service.

While the ACR application 139 b interacts with the ACR engine managementunit 137 b through the ACR application interface of the ACR middleware138 b, it may select the ACR engine 136 b to be used. For this, the ACRapplication interface that the ACR application 139 b uses is shown inTable 4 and Table 5 below.

TABLE 4 Name description Property readonly List of ACR enginesACRCollection installed in the Receiver acrCollection Method ACRgetACR(String Return an ACR object matched acrId) with ACR Id, orundefined if no item matched is found. The argument acrid means theunique identifier of the ACR engine/solution. ACRCollection Return thewhole set of ACR getAllACR( ) objects installed in the Receiver, orundefined if no ACR is present at the Receiver.

TABLE 5 Name description Property readonly Integer The number of ACRengines in length the collection Method ACR item(Integer Return the ACRobject at index) position index in the collection, or undefined if noitem is present at that position. The argument index means the index ofthe item that shall be returned.

While the ACR application 139 b interacts with the ACR engine 136 bthrough the ACR application interface of the ACR middleware 138 b, itcontrols the ACR engine 136 b, and receives content information orsignature from the ACR engine 136 b to additionally obtain an enhancedservice corresponding to the content information or signature. Then, theACR application 139 b plays them in the play device 120. For this, theACR application interface that the ACR application 139 b uses is shownin Table 6 below.

TABLE 6 Name description Property readonly String the unique id of theACR acrId engine readonly String name short name of ACR engine readonlyString Description of ACR engine description readonly Integer FP or WMacrType readonly Integer components used for ACR acrComponent (audio,video and both) readonly String for WM contentId readonly Date timingfor WM function onSignatureCaptured(String signature) ConstantACR_FINGERPRINT = 1 ACR done by fingerprint technology ACR_WATERMARK = 2ACR done by watermark technology ACR_AUDIO = 1 Audio component is usedfor the content recognition. ACR_VIDEO = 2 Video component is used forthe content recognition. ACR_AUDIO_VIDEO = 3 Both audio/video componentsare used. Method Boolean Initialize( ) Initialize the ACR object.Boolean Finalize( ) Destroy the ACR object. Boolean startACR( ) StartACR engine in order to get ready to content recognition. BooleanstopACR( ) Stop the ACR engine. Boolean for FP getSignature( ) Stringfor WM getContentId( ) String getTiming( ) for WM String getALSURL( )for WM Boolean setAudioAlgorithm (Integer audioNumCh, IntergetaudioSampleRate, Integer audioNumSamples, Integer audioBitPerSample)Boolean setVideoAlgorithm (Integer videoNumFrames, RECT screenRect)

Thus, one example of a fingerprint sample code of a content providerrelated application using such an ACR application interface is shown inTable 7.

TABLE 7 <HEAD> <title>NBCU app</title> <meta name=“acr ”content=“zeitera-fp-audio”/> <meta name=“appAttribute” content=“CP”/><object type=“application/x-acr” id=“ACRSol” / > <objecttype=“application/x-acrfactory” id=“ACRFac” / > <scripttype=“text/javascript”> var aspfpURL= “www.zeitera.com/fp-audio”; varacrSol = get ElementbyID(ACRSol); var acrFac = get ElementbyID(ACRFac);acrSol=acrFac.getACR(“zeitera-fp-audio”); if(acrSol == null) return;acrSol.Initiailize( ); while ( ) { // for FP  var signature =acrSol.getSignature( );  if (signature == null) continue;  var res =sendRequest(aspfpURL, signature); // 1st query   if (res.contentID.substring(0, 3) == “NBCU” ) { // CP matching success   var res2 = sendRequest(res.alsURL); // 2nd query    if (res2.asURL) {   // launch specific app application with this appURL    }   } }</script> </HEAD> <BODY>   ... </BODY>

Thus, one example of a fingerprint sample code of a content providerrelated application using such an ACR application interface is shown inTable 8.

TABLE 8   <HEAD> <title>NBCU Heros app</title> <meta name=“acr ”content=“zeitera-fp-audio”/> <meta name=“appAttribute”content=“Program”/> <object type=“application/x-acr” id=“ACRSol” / ><object type=“application/x-acrfactory” id=“ACRFac” / > <scripttype=“text/javascript”> var aspfpURL= “www.zeitera.com/fp-audio”; varacrSol = get ElementbyID(ACRSol); var acrFac = get ElementbyID(ACRFac);acrSol=acrFac.getACR(“zeitera-fp-audio”); if(acrSol == null)  return;acrSol.Initiailize( ); while ( ) { // for FP  var signature =acrSol.getSignature( );  if (signature == null) continue;  var res =sendRequest(aspfpURL, signature); // 1st query   if(res.contentID.substring(0, 9) == “NBCU-Heros” ) { // Program matchingsuccess    var res2 = sendRequest(res.alsURL, res.contentID,res.timing);    // 2nd query    if (res2.asURL) {    // launch specificapp application with this appURL    }   } } </script> </HEAD> <BODY></BODY>

Thus, one example of a watermarking sample code of a content providerrelated application using such an ACR application interface is shown inTable 9.

TABLE 9   <HEAD> <title>NBCU app</title> <meta name=“acr ”content=“nielsen-wm-audio”/> <meta name=“appAttribute” content=“CP”/><object type=“application/x-acr” id=“ACRSol” / > <objecttype=“application/x-acrfactory” id=“ACRFac” / > <scripttype=“text/javascript”> var aspfpURL= “www.nielsen.com/wm-audio”; varacrSol = get ElementbyID(ACRSol); var acrFac = get ElementbyID(ACRFac);acrSol=acrFac.getACR(“nielsen-wm-audio”); if(acrSol == null) return;acrSol.Initiailize( ); while ( ) { // for WM  var contentId =acrSol.getContentId( );  if (contentId == null) continue;  var alsURL =acrSol.getALSURL( );  if (contentId.substring(0, 3) == “NBCU” ) { // CPmatching success   var res = sendRequest(alsURL); // 2nd query   if(res.asURL) {    // launch specific app application with this appURL   } } } </script> </HEAD> <BODY>  ... </BODY>

Thus, one example of a watermarking sample code of a content providerrelated application using such an ACR application interface is shown inTable 10.

TABLE 10   <HEAD> <title>NBCU Heros app</title> <meta name=“acr ”content=“nielsen-wm-audio”/> <meta name=“appAttribute”content=“Program”/> <object type=“application/x-acr” id=“ACRSol” / ><object type=“application/x-acrfactory” id=“ACRFac” / > <scripttype=“text/javascript”> var aspfpURL= “www.nielsen.com/wm-audio”; varacrSol = get ElementbyID(ACRSol); var acrFac = get ElementbyID(ACRFac);acrSol=acrFac.getACR(“nielsen-wm-audio”); if(acrSol == null) return;acrSol.Initiailize( ); while ( ) { // for WM  var contentId =acrSol.getContentId( );  if (contentId == null) continue;  var timing =acrSol.getTiming( );  var alsURL = acrSol.getALSURL( );  if(contentId.substring(0, 9) == “NBCU-Heros” ) { // Program   matchingsuccess   var res = sendRequest(alsURL, contentId, timing); // 2nd query  if (res.asURL) {    // launch specific app application with thisappURL   }  } } </script> </HEAD> <BODY>  ... </BODY>

FIG. 14 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

Although the broadcast receiving device 60 and the video display device100 are turned on and the external input port 111 of the video displaydevice 100 is selected, the video display device 100 may not performautomatic content recognition before the ACR application 139 b isexecuted.

Once the corresponding channel related ACR application 139 b is executedby a user recognizing a channel of a main AV content, the ACRapplication 139 b queries and information on the ACR engine 136 b thatit uses to the ACR engine management unit 137 b and receives it inoperation S501.

Then, the ACR application 139 b starts the ACR engine 136 b that it usesin operation S503 and obtains content information through the ACR engine136 b in operation S505.

The ACR application 139 b obtains an available enhanced service relatingto a main AV content through the obtained content information inoperation S507, and displays a color button on the display unit 121 tonotify a user that an enhanced service is available in operation S509.

When the ACR application 139 b receives the use of the enhanced service,it plays the enhanced service in the play device 120 in operation S511.

When the ACR middleware 138 b receives a user input relating to thetermination of the ACR application 139 b, it terminates the ACRapplication 139 b in operation S513.

FIG. 15 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

As shown in FIG. 15, the enhanced service management unit 130 mayinclude an operating system 131 c, an audio driver 132 c, a video driver133 c, a network driver 134 c, an ACR platform API 135 c, an ACR engine136 c, an ACR middleware 138 c, and at least one ACR application 139 c.

When the enhanced service management unit 130 is used as shown in FIG.15, a user may need to necessarily recognize a channel or content.However, since the number of queries for ACR may be reduced, serveroverhead may be decreased.

The audio driver 132 c, the video driver 133 c, the network driver 134c, the ACR platform API 135 c, and the ACR middleware 138 c in theenhanced service management unit 130 of FIG. 15 are identical or similarto those 132 a, 133 a, 134 a, 135 a, and 138 a in the enhanced servicemanagement unit 130 FIG. 11. Therefore, their detailed descriptions willbe omitted.

The ACR engine 136 c extracts feature information from some frames or acertain section of audio samples of a main AV content to transmit aquery including the extracted feature information to an ACR adjustmentserver (not shown). The ACR adjustment server may transmit a fingerprintserver address in response to the query. Additionally, the ACRadjustment server may transmit content information corresponding to thefeature information together with the fingerprint server address.

When receiving the fingerprint server address from the ACR engine 136 c,the ACR application 139 c directly transmit a query, which includes thefeature information that the ACR engine 136 c extracts, to thefingerprint server corresponding to the fingerprint server address. Forthis, the ACR application interface that the ACR application 139 c usesis shown in Table 11 below.

TABLE 11 Name description Property Readonly String For fingerprint,signature function (String For fingerprint, signature) function pointer:pointed onSignatureCaptured function will return signature value fromlower ACREngine, event driven Method Boolean For fingerprint,getSignature( ) asynchronous call to get signature, will useonSignatureCaptured Boolean For watermark, getContentID( ) Returnrecognized content id, or undefined if content has not been recognized.Boolean When getSignature fails resetACREngine( ) several times(chchange), app will give control to ACREngine to query back toACRCoordinator. String Return ACRBackend URL getACRBackend( )

Thus, one example of a fingerprint sample code using such an ACRapplication interface is shown in Table 12.

TABLE 12   <HEAD> <title>CP app</title> <object type=“application/x-acr”id=“ACRSol” / > <script type=“text/javascript”> var acrSol = getElementbyID(ACRSol); acrSol.onSignatureCaptured = returnSignature; varacrBE = acrSol.getACRBackend( ); function doACR( ) {  acrSol.getSignature( );   setTimer(“acrTimeout”, 1000); } functionacrTimeout( ) {   timeout++;   if(timeout > 3)    acrSol.resetACREngine(); } function returnSignature(signature) {   timeout = 0;  doSpecificJob(signature); } function doSpecificJob(signature) {   //do specific work regarding this time } </script> </HEAD> <BODYonload=“doACR( )”> ... </BODY>

FIG. 16 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

When the broadcast receiving device 60 and the video display device 100are turned on and the external input port 111 of the video displaydevice 100 is selected, the ACR engine 136 c extracts featureinformation from some frames or a certain section of audio samples of amain AV content to transmit a query including the extracted featureinformation to an ACR adjustment server (not shown) in operation S601.

The ACR engine 136 c receives a fingerprint server address in responseto the query in operation S603.

The ACR application 139 c obtains the fingerprint server address fromthe ACR engine 136 c through a getACRBackend( ) method.

The ACR application 139 c obtains feature information of a certain videoframe section or a certain audio section of a main AV content from theACR engine 136 c through a getSignature( ) method in operation S607.

The ACR application 139 c transmits a query, which includes the obtainedfeature information, to a fingerprint server corresponding to theobtained fingerprint server address in operation S609.

The ACR application 139 c receives content information in response tothe query in operation S611.

The ACR application 139 c obtains an available enhanced service on thebasis of the obtained content information in operation S613, anddisplays a color button on the display unit 121 to notify a user that anenhanced service is available in operation S615.

When the ACR application 139 c receives the use of the enhanced service,it plays the enhanced service in the play device 120 in operation S617.

When the ACR middleware 138 c receives a user input relating to thetermination of the ACR application 139 c, it terminates the ACRapplication 139 c in operation S619.

FIG. 17 is a hierarchy diagram of an enhanced service management unitaccording to an embodiment.

As shown in FIG. 17, the enhanced service management unit 130 mayinclude an operating system 131 d, an audio driver 132 d, a video driver133 d, a network driver 134 d, an ACR platform API 135 d, at least oneACR engine 136 d, an ACR engine management unit 137 d, an applicationmanagement unit 138 d, and at least one ACR application 139 d.

When the enhanced service management unit 130 is used as shown in FIG.17, a user doesn't need to necessarily recognize a channel or contentand an ACR operation may be performed always.

The audio driver 132 d, the video driver 133 d, the network driver 134d, the ACR platform API 135 d, and the plurality of ACR engines 136 d inthe enhanced service management unit 130 of FIG. 17 are identical orsimilar to those 132 a, 133 a, 134 a, 135 a, and 136 a in the enhancedservice management unit 130 FIG. 11. Therefore, their detaileddescriptions will be omitted.

The ACR engine management unit 137 d confirms that the ACR engine 136 dmatches to the main AV content received from the external input port 111and confirms that the ACR engine 136 d matches to the application 139 d.At this point, the ACR engine management unit 137 d controls theplurality of ACR engines 136 d to simultaneously or sequentially performwatermark extraction or query transmission. The ACR engine managementunit 137 d controls the plurality of ACR engines 136 d through priority.If one of the plurality of ACR engines 136 d is successful inrecognizing a main AV content, the ACR engine management unit 137 d putsthe remaining ACR engines in sleep mode and obtains the contentinformation of the main AV content from the operating ACR engine.

While the application management unit 138 d interacts with the ACRengine management unit 137 d, it controls generation, start, stop,sleep, and termination of the ACR application.

The ACR application 139 d may be a built-in application. The ACRapplication 139 d may control the ACR engine 136 b through functionsshown in Table 13.

TABLE 13 Name Function char *getContentID( ) calls Date getTiming( )char *getSignature( ) Boolean   registerEventListener(EvtType   evtType,Boolean (*eventListener)(EvtType evtType, char *contentId, Date *timing,char *signature)); Data struct Date { Structure   int  year;   int month;   int  day;   int  hour;   int  minute;   int  second;   int msecond; }; typedef enum EvtType  {   EVT_PROGRAM_CHANGED,  EVT_CHANNEL_CHANGED,   EVT_ALL };

FIG. 18 is a flowchart illustrating an operation of an enhanced servicemanagement unit according to an embodiment.

When the broadcast receiving device 60 and the video display device 100are turned on and the external input port 111 of the video displaydevice 100 is selected, the ACR engine management unit 137 d confirmsthat the ACR engine 136 a matches to the main AV content received fromthe external input port 111 in operation S701. At this point, the ACRengine management unit 137 d controls the plurality of ACR engines 136 dto simultaneously or sequentially perform watermark extraction or querytransmission. The ACR engine management unit 137 d controls theplurality of ACR engines 136 d through priority.

If one of the plurality of ACR engines 136 d is successful inrecognizing a main AV content, the ACR engine management unit 137 d putsthe remaining ACR engines in sleep mode in operation S702, and obtainsthe content information of the main AV content from the operating ACRengine in operation S703.

Then, the ACR engine management unit 137 d confirms an availableapplication relating to content information in operation S704, andrequests the execution of the confirmed application 139 d to theapplication management unit 138 d.

The application management unit 138 d executes the confirmed application139 d in background in operation S705.

The application 139 d executed in background displays a color button onthe display unit 121 to notify a user that enhanced service is availablein operation S707.

When the application 139 d executed in background receives the use ofthe enhanced service from a user, it plays the enhanced service in theplay device 120 in operation S709.

If a channel or main AV content is changed in operation S710, the ACRengine management unit 137 d confirms the ACR engine 136 a matching tothe application 139 d and connects the confirmed ACR engine 136 a to theapplication 139 d in operation S711. At this point, the ACR enginemanagement unit 137 d controls the plurality of ACR engines 136 d tosimultaneously or sequentially perform watermark extraction or querytransmission.

If the application management unit 138 d receives a user input for thetermination of the application 139 d in operation S712, it terminatesthe application 139 d in operation S713.

Then, referring to FIGS. 19 and 20, a method of synchronizing a playtime of a main AV content with a play time of an enhanced serviceaccording to an embodiment will be described.

Then, referring to FIGS. 19 and 20, a method of synchronizing a playtime of a main AV content with a play time of an enhanced serviceaccording to an embodiment will be described.

Enhanced service information may include a start time of an enhancedservice. At this point, the video display device 100 may need to startthe enhanced service at the start time. However, since the video displaydevice 100 receives a signal transmitting an uncompressed main AVcontent with no time stamp, the reference time of a plying time of themain AV content is different from that of a start time of the enhancedservice. Although the video display device 100 receives a main AVcontent having time information, the reference time of a plying time ofthe main AV content may be different from that of a start time of theenhanced service, like rebroadcasting. Accordingly, the video displaydevice 100 may need to synchronize the reference time of the main AVcontent with that of the enhanced service. Especially, the video displaydevice 100 may need to synchronize the play time of the main AV contentwith the start time of the enhanced service.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S801. The section of the mainAV content may include at least one of some video frames or a certainaudio section of the main AV content. Time that the enhanced servicemanagement unit 130 extracts the section of the main AV content isdesignated as Tn.

The enhanced service management unit 130 obtains content information ofa main AV content on the basis of the extracted section. In more detail,the enhanced service management unit 130 decodes information encodedwith invisible watermark in the extracted section to obtain contentinformation. Additionally, the enhanced service management unit 130 mayextract feature information in the extracted section, and obtain thecontent information of the main AV content from the fingerprint server22 or the content information storage unit 151 on the basis of theextracted feature information. Time that the enhanced service managementunit 130 obtains the content information is designated as Tm.

Moreover, the content information includes a start time Ts of theextracted section. After the content information acquisition time Tm,the enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of the enhanced service on thebiases of Ts, Tm, and Tn. In more detail, the enhanced servicemanagement unit 130 regards the content information acquisition time Tmas a time Tp calculated by the following Equation 1.

Tp=Ts+(Tm−Tn)  [Equation 1]

Additionally, the enhanced service management unit 130 regards a time ofwhen Tx elapses after the content information acquisition time as Tp+Tx.

Then, the enhanced service management unit 130 obtains an enhancedservice and its start time Ta on the obtained content information inoperation S807.

If the synchronized play time of the main AV content is identical to thestart time Ta of the enhanced service, the enhanced service managementunit 130 starts the obtained enhanced service in operation S809. In moredetail, the enhanced service management unit 130 may start the enhancedservice when the following Equation 2 is satisfied.

Tp+Tx=Ta  [Equation 2]

FIG. 20 is a conceptual diagram illustrating a method of synchronizing aplay time of a main AV content with a play time of an enhanced serviceaccording to an embodiment.

As shown in FIG. 20, the video display device 100 extracts an AV sampleduring a system time Tn.

The video display device 100 extracts feature information from theextracted AV sample, and transmits a query including the extractedfeature information to the fingerprint server 22 to receive a queryresult. The video display device 100 confirms whether a start time Ts ofthe extracted AV sample corresponds to 11000 ms at Tm by parsing thequery result.

Accordingly, the video display device 100 regards the time of when thestart time of the extracted AV sample is confirmed according to Equation1 as Ts+(Tm−Tn), so that, after that, the play time of the main AVcontent may be synchronized with the start time of the enhanced service.

Next, an ACR application will be described according to variousembodiments with reference to FIGS. 21 to 50.

The ACR application may include an auto playing application, a contentprovider application, and an video display manufacturer application.

FIG. 21 is a screen when an auto playing application is executedaccording to an embodiment.

The auto playing application automatically plays an enhanced serviceeven when a user makes no action. In this sense, the auto playingapplication is designated as a lean back application in thisspecification.

The auto playing application automatically plays an enhanced servicesuch as scene related information that a content provider orbroadcasting station provides during playing of the main AV contentaccording to a play time of the main AV content, without a user's input.That is, a content provider or broadcasting station entirely controlsthe auto playing application.

As shown in FIG. 21, once the auto playing application is executed, thescreen 200 may include a sub screen 210 where a reduced main AV contentis played and a sub screen 220 where additional information isdisplayed. The sub screen 200 where additional information is displayedmay be spatially separated from the sub screen 210 where a main AVcontent is played.

Moreover, when an auto playing application is executed, an area whereadditional information is displayed may be semi-transparent. In thiscase, the area where additional information is displayed may overlay onan area where a main AV content played.

FIG. 22 is a screen when a content provider application is executedaccording to an embodiment.

In this specification, the content provider application is designated asa full interactive application.

As shown in FIG. 22 (A), the video display device 100 downloads andstores a plurality of applications. FIG. 22 (A) illustrates icons 230 ofthe applications. A content provider application among the plurality ofapplications is an application created and distributed by a contentprovider, and thus, may be under control of the content provider.

A user of the video display device 100 may download a content providerapplication from an application store and determines whether to executethe downloaded content provider application.

FIG. 22 (B) is a screen when a content provider application is executed.As shown in FIG. 22 (B), the screen 200 may include the sub screen 210where a reduced main AV content is played and the sub screen 220 whereadditional information is displayed.

Hereinafter, a video display device manufacturer application will bedescribed according to an embodiment with reference to FIGS. 23 to 50.

The video display device manufacturer application is created andcontrolled by a video display device manufacturer, and may be stored inadvance when the video display device 100 is manufactured. That is, thevideo display device manufacturer application may provide the sameservice and user interface regardless of a content provider or content.

The video display device manufacturer application may include an areawhere an enhanced service that a content provider provides is displayedand an area where an enhanced service that a video display devicemanufacturer provides is displayed, separately. A user of the videodisplay device 100 may determine whether to execute the video displaydevice manufacturer application.

The video display device manufacturer application may have two types.

The video display device manufacturer application of the first type hasa resizing user interface. The video display device manufacturerapplication having the resizing user interface reduces the size of anarea where a main AV content is displayed to display all the main AVcontent in a reduced area. Additionally, the video display devicemanufacturer application having the resizing user interface reshapes anarea where an application is displayed with an L or inverse-L shapedstructure.

The video display device manufacturer application of the second type hasan overlay user interface. The video display device manufacturerapplication having the overlay user interface maintains the size of anarea where a main AV content is displayed, and overlaps an area where anapplication is displayed on the main AV content. Since the area where anapplication is displayed covers the main AV content, an application areamay be semi-transparent.

FIG. 23 is a screen when a video display device manufacturer applicationhaving a resizing user interface is executed according to an embodiment.

As shown in FIG. 23, the screen 300 of when a video display devicemanufacturer application having a resizing user interface is executedincludes a main content area 310 where a main AV content is displayed, amain menu area 320 where a main menu is displayed, a notification area330 where a notification message is displayed, a control button area 340where a control button is displayed to control the video display devicemanufacturer application, a sub menu area 350 where a sub menu isdisplayed, an additional information area 360 where additionalinformation is displayed, and an advertisement area 370 whereadvertisement is displayed.

The control button area 340 includes an ending button 341 for ending thevideo display device manufacturer application, a hiding button 342 forhiding the video display device manufacturer application, and a backbutton 343 for returning to previous information.

In the video display device manufacturer application having the resizinguser interface, the main AV content area 310 is disposed on the top-leftof the screen 300; the main menu area 320 and the notification area 330is disposed on the bottom of the main AV content area 310; the controlbutton area 340 is disposed on the top-left; the sub menu area 350 isdisposed between the control button area 340 and the additionalinformation area 360; the advertisement area 370 is disposed on thebottom-right; and the additional information area 360 is disposedbetween the sub menu area 350 and the advertisement area 370. However,their positions may vary.

FIG. 24 is a screen when a video display device manufacturer applicationhaving an overlay user interface is executed according to an embodiment.

As shown in FIG. 24, like the video display device manufacturerapplication having the resizing user interface, the screen 300 of whenthe video display device manufacturer application having the overlayuser interface is executed includes a main content area 310 where a mainAV content is displayed, a main menu area 320 where a main menu isdisplayed, a notification area 330 where a notification message isdisplayed, a control button area 340 where a control button is displayedto control the video display device manufacturer application, a sub menuarea 350 where a sub menu is displayed, an additional information area360 where additional information is displayed, and an advertisement area370 where advertisement is displayed.

In the video display device manufacturer application having the overlayuser interface, the main AV content area 310 is disposed on the top-leftof the screen 300; the main menu area 320 and the notification area 330is disposed on the bottom of the main AV content area 310; the controlbutton area 340 is disposed on the top-left; the sub menu area 350 isdisposed between the control button area 340 and the additionalinformation area 360; the advertisement area 370 is disposed on thebottom-right; and the additional information area 360 is disposedbetween the sub menu area 350 and the advertisement area 370. However,their positions may vary.

Hereinafter, a video display device manufacturer application having aresizing user interface will be mainly described.

The video display device manufacturer application receives necessaryinformation from an ACR engine built in the video display device 100 andhas two types of menus largely according to its character.

The first type menu is for a time insensitive enhanced service. That is,when a menu relating to the time insensitive enhanced service isselected, the video display device 100 obtains an identifier of a mainAV content that is currently played on a screen, obtains an enhancedservice on the basis of the obtained identifier, and displays theobtained enhanced service on the additional information area 360.Examples of such a time insensitive enhanced service includes titles,genres, actors, directors, and other people's evaluations, short clips,and short explanations. Through this basic information, the videodisplay device 100 may search additional information, link to a socialnetwork service such as Facebook and Twitter, or provide a recommendedservice and a related advertisement.

The second type menu is for a time sensitive enhanced service. That is,when a menu relating to the time sensitive enhanced service is selected,the video display device 100 obtains an identifier and play timeinformation of a main AV content that is currently played on a screen,obtains a time sensitive enhanced service on the basis of the obtainedidentifier and play time information, and displays the obtained enhancedservice on the additional information area 360.

The main menu of the video display device manufacturer applicationincludes a program menu, an information menu, a social menu, arecommendation menu, and a catalogue menu. Among these, the timeinsensitive menu includes the information menu, the social menu, therecommendation menu, and the catalogue menu, and the time sensitive menuincludes the program menu. If the time insensitive menu is selected, thetime sensitive application may be called. Other than these five mainmenus, other main menus may be added. If each main menu is selected, asub menu of the selected main menu is displayed on the sub menu area350.

FIG. 25 is a flowchart illustrating a method of executing a videodisplay device manufacturer application according to an embodiment.

When receiving a user input corresponding to an application selectionmenu display through pressing of a specific key of a remote controller,shaking of a motion remote controller in a downward direction, zoomingof the cursor of a motion remote controller in the bottom direction ofthe screen while a main AV content is played in operation S901, thevideo display device 100 displays an application selection menu of FIG.26 in operation S903.

FIG. 26 is a screen when an application selection menu is displayedaccording to an embodiment.

As shown in FIG. 26, the application selection menu 380 is disposed atthe bottom of the screen 300. The application selection menu 380 may besemi-transparent.

The application selection menu 380 includes execution icons of aplurality of applications installed in the video display device 100. Oneicon of the displayed execution icons corresponds to the execution iconof the video display device manufacturer application.

When receiving a user input corresponding to an icon selection of thevideo display device manufacturer application through manipulating ofthe left and right arrow keys in a remote controller or moving of thecursor according to a movement of a motion remote controller inoperation S905, the video display device 100 executes the video displaydevice manufacturer application in operation S907. Through this, a usermay immediately execute the application if he/she wants without accessto an application store while watching a main AV content.

Once the video display device manufacturer application is executed, itsmain menu is displayed. If a program menu is selected by default, itssub menu is displayed on the sub menu area 350. An all sub menu isselected by default among the sub menus of the program menu. Focus ismade on the all sub menu. As time elapses, an enhanced service displayedon the additional information area 360 is automatically updated. The <-button is used for returning to a previous menu and the X button is usedfor returning to an entire screen after closing the video display devicemanufacturer application.

Then, a video display device manufacturer application will be describedaccording to an embodiment with reference to FIGS. 27 to 29.

The program main menu is a menu for displaying an application, which istime-related to a main AV content that is being played, on theadditional information area 360. Enhanced service information fordisplaying an application linking with the main AV content may beprovided from a content owner such as TV broadcasters. The video displaydevice 100 analyzes the enhanced service information to display acorresponding application in the additional information area 360 at theright timing. UI in a screen is shown in the drawing below.

The program menu includes an all sub menu, a participation sub menu, anda discovery sub menu.

The all sub menu is a menu for displaying all applications related tothe participation sub menu and discovery sub menu on the additionalinformation area 360. The participation sub menu is a menu fordisplaying an application inducing program participation such as votingin a program. The discovery sub menu is a menu for displaying anapplication such as scene related information. The sub menu of theprogram menu may be added or edited according to contents of anapplication that a content owner wants.

FIG. 27 is a screen when an all sub menu of a program main menu in avideo display device manufacturer application is selected according toan embodiment.

When the all sub menu, i.e., the first sub menu of the program mainmenu, is selected, the video display device 100 displays allapplications related to the participation sub menu and display sub menuon the additional information area 360.

When the all sub menu is selected, the video display device 100 displaysan application or event synchronized with a specific time on theadditional information area 360. If there is no application synchronizedwith a specific time, the video display device 100 may leave theadditional information area 350 as empty space. However, since the emptyspace may not give motivation for additional interaction to a user, ifthere is no application synchronized with a specific time, the videodisplay device 100 may gather applications or events that occur prior tonow in order to display a history.

When a user enters into the video display device manufacturerapplication, it may be seen that the user may have an intention toperform an interactive application related to a main AV content whilewatching the main AV content. While navigating several menus, a user maynot watch an event at a specific time. Especially, when a user navigatesmenus for time insensitive enhanced services, the above case may likelyoccur. Therefore, after entering into the video display devicemanufacturer application, if a time sensitive event occurs duringnavigation, the video display device 100 displays a notification messagefor this on the notification area 330 to notify a user that there is aspecial event at the current time broadcasted from a broadcastingstation.

While a user confirms music information or place information through thevideo display device manufacturer application, a time linked applicationoccurs, the video display device 100 displays a notification message onthe notification area 330. When receiving a user input corresponding tothe Go To button press, the video display device 100 enters into a submenu related to an occurring time linked application and displays theoccurring time linked application on the additional information area360. The notification message may be a text message. Additionally, thevideo display device 100 may display an amount of the deliveredapplication as a number. The notification message may be delivered to aviewer in a Push method.

FIG. 28 is a screen when a participation sub menu of a program main menuin a video display device manufacturer application is selected accordingto an embodiment.

As shown in FIG. 28, once the participation sub menu is selected, thevideo display device 100 displays an application inducing viewerparticipation such as voting on the additional information area 360. Ifa viewer participation application occurs, the video display device 100displays it on the additional information area 360. If the viewerparticipation application is voting, the video display device 100 maydisplay voting questions and voting results. Moreover, the video displaydevice 100 displays current events and also previous events that theviewer participates on the bottom of the current event, for the viewer'sconvenience.

FIG. 29 is a screen when a discovery sub menu of a program main menu ina video display device manufacturer application is selected according toan embodiment.

As shown in FIG. 29, once the discovery sub menu is selected, the videodisplay device 100 displays all time linked applications besides aviewer participation application on the additional information area 360.For example, the video display device 100 displays scene relatedinformation on products or places shown in a main AV content. The videodisplay device 100 may display current scene related information andprevious scene related information simultaneously.

Moreover, the video display device 100 may display a notificationmessage that there is a time linked application or event on a sub menu.For example, if a viewer participation application occurs, the videodisplay device 100 displays an icon having a flashing N on theparticipation sub menu to induce viewer participation. If a user doesnot respond to the notification for a predetermined time, the videodisplay device 100 may remove the notification.

Hereinafter, an information main menu of a video display devicemanufacturer application will be described according to an embodimentwith reference to FIGS. 30 to 36.

The information main menu corresponds to a time insensitive area. Likethe UI of the program main menu, the information main menu does notoperate in a UI manner that an application is automatically displayedand disappears at a specific time. Of course, although theabove-mentioned time linked application may be displayed through thismenu, when a main menu other than the program main menu is selected, asecond service such as search service or Twitter service may be linked.

The information main menu includes a search sub menu, a Twitter submenu, and a news sub menu.

Then, a search sub menu of an information main menu in a video displaydevice manufacturer application will be described according to anembodiment with reference to FIGS. 30 to 32.

FIG. 30 is flowchart illustrating a searching method according to anembodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection.

Once the search sub menu of an information main menu in a video displaydevice manufacturer application is selected in operation S1002, thevideo display device 100 obtains a search list including a plurality ofsearch items corresponding to an enhanced service on the basis of theobtained content information in operation S1003. The search listincludes at least one of genres, actors, directors, places, andproducts.

The video display device 100 displays the search list on the additionalinformation area 360 in operation S1007. A display example of the searchlist will be described with reference to FIG. 30.

FIG. 31 is a screen when a search sub menu of an information main menuin a video display device manufacturer application is selected accordingto an embodiment.

As shown in FIG. 31, the search list includes characters of a main AVcontent. The video display device 100 may display at least one ofpictures or names of a search item on the additional information area360.

When one of a plurality of search items is selected through the cursorof a remote controller in operation S1009, the video display device 100searches information on the selected search item in operation S1011. Atthis point, the video display device 100 may use an internet searchengine such as Google. Moreover, the video display device 100 may useinformation such as Tribune media or IMDb.

The video display device 100 displays the searched information on theadditional information area 360 in operation S1013. Since the searchedresult is automatically retrieved through programming such as Open APIof a service provider, time for a screen design may be greatly saved. Adisplay example of the search information will be described withreference to FIG. 32.

FIG. 32 is another screen when a search sub menu of an information mainmenu in a video display device manufacturer application is selectedaccording to an embodiment.

As shown in FIG. 32, the video display device 100 displays a searchresult for the selected search item on the additional information area360. Moreover, the video display device positions a Twitter button andan edit button on the additional information area 360 to follow anactor's Twitter or edit an actor's profile such as filmography orbiography.

Like this, if the searching method according to an embodiment is used,information related to a main AV data may be searched through one click,so that an effort to input a search word may be reduced.

Next, a Twitter sub menu of an information main menu in a video displaydevice manufacturer application will be described according to anembodiment with reference to FIGS. 33 to 34.

FIG. 33 is flowchart illustrating a chatting window displaying methodaccording to an embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1101.

Once a Twitter sub menu of an information main menu in a video displaydevice manufacturer application is selected in operation S1102, thevideo display device 100 obtains a Twitter chatting list correspondingto an enhanced service on the basis of the obtained content informationin operation S1103. The video display device 100 writes a program nameon Hashtag of Twitter to load a Twitter chatting list related to theprogram. This chatting list is linked to Twitter and is continuouslyupdated in real time.

The video display device 100 displays one chatting window in the Twitterchatting list on the additional information area 360 through a userselection or automatically in operation S1107. A display example of thechatting window will be described with reference to FIG. 34.

FIG. 34 is a screen when a Twitter sub menu of an information main menuin a video display device manufacturer application is selected accordingto an embodiment. 6

As shown in FIG. 34, the video display device 100 displays a chattingwindow on the additional information area 360 to support chattingbetween viewers watching the same program.

When a program is broadcasted, a user of the program may directly writea message in a chatting room actually. If the video display device 100has a log in mechanism like Smart TV, a user of the video display device100 may immediately start chatting with his own ID in a logged in statewhen entering into a menu. If it is inconvenient to make an input to thevideo display device 100 through a remote controller, the video displaydevice 100 may display only a chatting message. If the video displaydevice 100 is linked to a second device such as Smart Phone or tablet, amessage may be typed using the input device of the second device.Although information in Hashtag is a program name basically, it may beselected as a tag of actors or directions, i.e., the above-mentionedanother basic information, so that a user may selectively participate inchatting of a corresponding subject.

Next, a news sub menu of an information main menu in a video displaydevice manufacturer application will be described according to anembodiment with reference to FIGS. 35 to 36.

FIG. 35 is a flowchart illustrating a news obtaining method according toan embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1201.

Once the news sub menu of an information main menu in a video displaydevice manufacturer application is selected in operation S1202, thevideo display device 100 obtains news corresponding to an enhancedservice on the basis of the obtained content information in operationS1203. At this point, the news may or may not relate to a main AVcontent. Or, the news may be provided from a broadcaster of a main AVcontent. If the broadcaster does not provide news through Open API, thevideo display device 100 may receive news through News feeds of aninternet service such as Google or Yahoo.

The video display device 100 displays the obtained news on theadditional information area 360 in operation S1207. A display example ofnews will be described with reference to FIG. 36.

FIG. 36 is a screen when a news sub menu of an information main menu ina video display device manufacturer application is selected according toan embodiment.

As shown in FIG. 36, the video display device 100 may display theobtained news on the additional information area 360.

Hereinafter, a social main menu of a video display device manufacturerapplication will be described according to an embodiment with referenceto FIGS. 37 to 42.

The social main menu corresponds to a time insensitive area. Like the UIof the program main menu, the social main menu does not operate in a UImanner that an application is automatically displayed and disappears ata specific time. Of course, although the above-mentioned time linkedapplication may be displayed through this menu, when a main menu otherthan the program main menu is selected, a second service such asFacebook may be linked.

The social main menu includes a viewer list sub menu and a preferencedisplay sub menu.

Then, a viewer list sub menu of a social main menu in a video displaydevice manufacturer application will be described according to anembodiment with reference to FIGS. 37 to 40.

FIG. 37 is flowchart illustrating a viewer list displaying methodaccording to an embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1301.

Once a viewer list sub menu of a social main menu in a video displaydevice manufacturer application is selected in operation S1302, thevideo display device 100 obtains information on a friends list andwhether each friend watches a main AV content on the basis of theobtained content information in operation S1303. The video displaydevice 100 may be linked with a social network service such as Facebookto obtain a list of friends watching a main AV content.

The video display device 100 displays the obtained viewer list on theadditional information area 360 in operation S1307. A display example ofthe viewer list will be described with reference to FIG. 38.

FIG. 38 is a screen when a viewer list sub menu of a social main menu ina video display device manufacturer application is selected according toan embodiment.

As shown in FIG. 38, the video display device 100 may display thepictures and names of the viewers corresponding to the obtained viewerlist on the additional information area 360.

Especially, the video display device 100 may display a friends listtogether with whether each friend watches contents or only a list offriends watching contents currently.

When one friend is selected from the displayed friends list in operationS1309, the video display device 100 confirms whether the selected friendwatches a main AV content in operation S1311.

If the selected friend watches the main AV content, the video displaydevice 100 displays the chatting window of the selected friend on theadditional information area 360 in operation S1313.

Additionally, if the selected friend does not watch the main AV content,the video display device 100 sends a message recommending watching themain AV content to the e-mail or mobile phone of the selected friend inoperation S1315. Even if the selected friend does not watch the main AVcontent, the video display device 100 may display the chatting window ofthe selected friend on the additional information area 360.

For chatting, a user may use the remote controller of the video displaydevice 100 or the input device of the second device connected to thevideo display device 100, and may chat with friends through a socialnetwork service application of a smart phone regardless of the videodisplay device 100. In any cases, a chatting screen of the socialnetwork service may be displayed on the screen of the video displaydevice 100. Through buttons for evaluating a program, a user may sendhis/her intention to friends. For example, a user may notify friendsthat he/she currently watches a main AV content and may recommendfriends who do not watch the main AV content currently to watch itthrough a “good” button.

Like this, the video display device 100 automatically realizes who iswatching which program by using an ACR technique through a viewer listsub menu. In order to confirm who is watching which program, a userneeds to log in. Once an account for log in is registered in the videodisplay device 100, each time the video display device 100 is turned on,a user may log in automatically. Therefore, the video display device 100may realize who is watching a main AV content.

FIG. 39 is a signal flow when a video display device obtains a friendslist and watching information according to an embodiment.

As shown in FIG. 39, if a video display device of a friend A plays amain AV content A, it transmits an identifier of the main AV content Aand an identifier of the friend A to the SNS providing server 75 inoperation S1401.

Additionally, if a video display device of a friend B plays the main AVcontent A, it transmits an identifier of the main AV content A and anidentifier of the friend B to the SNS providing server 75 in operationS1403.

If the video display device 100 plays the main AV content A, ittransmits a user ID and an identifier of the main AV content A to theSNS providing server 75 in operation S1405.

The SNS providing server 75 confirms information on a friends list ofthe video display device 100 and whether each friend watches a main AVcontent on the basis of the friend relationship information and receivedcontent identifiers in operation S1407.

FIG. 40 is a signal flow when a video display device obtains a friendslist and watching information according to another embodiment.

As shown in FIG. 40, if a video display device of a friend A plays amain AV content A, it transmits an identifier of the friend A to the SNSproviding server 75 in operation S1501, and transmits the main AVcontent A to the ACR sever in operation S1503. The ACR server 77 may beone of a watermark server 21, a fingerprint server 22, an enhancedservice information providing server 40, and an enhanced serviceproviding sever 50.

Additionally, if a video display device of a friend B plays the main AVcontent A, it transmits an identifier of the friend B to the SNSproviding server 75 in operation S1505, and transmits the main AVcontent A to the ACR sever 77 in operation S1507.

The video display device 100 transmits a user ID to the SNS providingserver 75 in operation 1509.

Then, the SNS providing server 75 provides a friend list correspondingto the user ID to the ACR server 75 in operation S1511.

Moreover, if the video display device 100 plays the main AV content A,it transmits a user ID and an identifier of the main AV content A to theACR server 75 in operation S1513.

The ACR server 75 confirms information on the friends list of the videodisplay device 100 and whether each friend watches a main AV content onthe basis of the friends list corresponding to the user ID and contentidentifier, and then, provides the confirmed information to the videodisplay device 100 in operation S1515.

Then, a preference display sub menu of a social main menu in a videodisplay device manufacturer application will be described according toan embodiment with reference to FIGS. 41 to 42.

FIG. 41 is flowchart illustrating a preference displaying methodaccording to an embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1601.

Once a preference display sub menu of a social main menu in a videodisplay device manufacturer application is selected in operation S1602,the video display device 100 obtains a preference display list includinga plurality of preference display items on the basis of the obtainedcontent information in operation S1603. Here, the preference displayitem may include preview or a preview address. The preview may be animage or a short video. The preview may be directly provided from abroadcaster through Open API, or may be obtained by a search engine ofan internet service, or obtained by extracting only a necessary portionfrom an internet server of a broadcaster. Images corresponding to corescenes of a time interval corresponding to several minutes may beprovided or one image for one program may be provided.

The video display device 100 displays the obtained preference displaylist on the additional information area 360 in operation S1607. Adisplay example of the viewer list will be described with reference toFIG. 42.

FIG. 42 is a screen when a preference display sub menu of a social mainmenu in a video display device manufacturer application is selectedaccording to an embodiment.

As shown in FIG. 42, the video display device 100 may display previewsin a preference display list and “Like it” buttons for preferencedisplay on the additional information area 360.

Once the “Like it” button corresponding to one preference display itemis selected in operation S1609, the video display device 100 posts apreview and simple text corresponding to the selected preference displayitem on SNS such as Facebook in operation S1611. Here, the simple textmay be automatically generated, or may be inputted through a remotecontroller or an input device of a second device.

Hereinafter, a recommendation main menu of a video display devicemanufacturer application will be described according to an embodimentwith reference to FIGS. 43 to 46.

The recommendation main menu corresponds to a time insensitive area.Like the UI of the program main menu, the recommendation main menu doesnot operate in a UI manner that an application is automaticallydisplayed and disappears at a specific time. Of course, through thismenu, the above-mentioned time linked application may be displayed butif a recommendation menu is selected, another service may be linked.

The recommendation main menu may include a previous episode sub menu, aVOD sub menu, and an application sub menu.

FIG. 43 is flowchart illustrating a digital media downloading methodaccording to an embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1701.

Once a recommendation main menu of a video display device manufacturerapplication is selected in operation S1702, the video display devicemanufacturer application obtains a digital media list on the basis ofthe obtained content information in operation S1703. If the previousepisode sub menu is selected, the video display device 100 obtains aprevious episode list. If the VOD sub menu is selected, the videodisplay device 100 obtains a VOD list that relates to a main AV contentor a provider of the main AV content recommends. If the application submenu is selected, the video display device 100 obtains a VOD list thatrelates to a main AV content or a provider of the main AV contentrecommends. Here, the digital media list may include a preview, apreview address, and an icon related image.

The video display device manufacturer application displays the obtaineddigital media list on the additional information area 360 in operationS1707.

An example of a previous episode list is described with reference toFIG. 44.

FIG. 44 is a screen when a previous episode sub menu of a recommendationmain menu in a video display device manufacturer application is selectedaccording to an embodiment.

As shown in FIG. 44, the video display device 100 displays previews andtitles of a previous episode list on the additional information area360. Like this, according to an embodiment, the video display device 100may automatically search and display a previous episode without user'smanipulation.

A display example of the VOD list will be described with reference toFIG. 45.

FIG. 45 is a screen when a VOD sub menu of a recommendation main menu ina video display device manufacturer application is selected according toan embodiment.

As shown in FIG. 45, the video display device 100 displays previews andtitles of a VOD list on the additional information area 360. Like this,according to an embodiment, the video display device 100 mayautomatically search and display VOD related to a main AV contentwithout user's manipulation.

A display example of the application list will be described withreference to FIG. 46.

FIG. 46 is a screen when an application sub menu of a recommendationmain menu in a video display device manufacturer application is selectedaccording to an embodiment.

As shown in FIG. 46, the video display device 100 displays icons andapplication names of the application list on the additional informationarea 360. Like this, according to an embodiment, the video displaydevice 100 may automatically search and display applications related toa main AV content without user's manipulation.

Once one item is selected from the digital media list in operationS1709, the video display device manufacturer application downloads adigital media corresponding to the selected item in operation S1711. Atthis point, the video display device 100 may download the digital mediaafter payment. If the selected item is a previous episode or VOD, thevideo display device 100 may play the downloaded digital media. If theselected item is an application, the video display device mayautomatically play the downloaded application or may play it through auser's selection.

Like this, according to an embodiment, the video display device maydownload digital media through simple manipulation.

Hereinafter, a catalog main menu of a video display device manufacturerapplication will be described according to an embodiment with referenceto FIGS. 47 to 50.

The catalog main menu corresponds to a time insensitive area. Like theUI of the program main menu, the catalog main menu does not operate in aUI manner that an application is automatically displayed and disappearsat a specific time. Of course, through this menu, the above-mentionedtime linked application may be displayed but if a catalog menu isselected, another service may be linked.

The catalog main menu may include a style sub menu, a music sub menu,and another sub menu.

FIG. 47 is flowchart illustrating a catalog displaying method accordingto an embodiment.

The video display device 100 extracts a certain section of a main AVcontent and obtains content information on the basis of the extractedsection in operation S1801.

Once a sub menu of a catalog main menu in a video display devicemanufacturer application is selected in operation S1802, the videodisplay device manufacturer application obtains a catalog list on thebasis of the obtained content information in operation S1803. If thestyle sub menu is selected, the video display device 100 obtains aclothing catalog list that relates to a main AV content or a provider ofthe main AV content recommends. If the music sub menu is selected, thevideo display device 100 obtains a music catalog list that relates to amain AV content or a provider of the main AV content recommends. If theother sub menu is selected, the video display device 100 obtains aproducts or places catalog list that relates to a main AV content or aprovider of the main AV content recommends. Here, the digital media listmay include a related image, price information, description, and coupon.The coupon may be numbers or QR codes. The video display device 100 maydisplay coupons only when a user watches a main AV content, or may allowa user to use it during the available period after storing it on thememory 150.

The video display device manufacturer application displays the obtainedcatalog list on the additional information area 360 in operation S1807.

A display example of the clothing catalog list will be described withreference to FIG. 48.

FIG. 48 is a screen when a style sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

As shown in FIG. 48, the video display device 100 displays makerinformation, price information, of description information of clothingrelated to a main AV content on the additional information area 360.Like this, according to an embodiment, the video display device 100 mayautomatically search and display a clothing catalog related to a main AVcontent without user's manipulation.

An example of a music catalog list is described with reference to FIG.49.

FIG. 49 is a screen when a music sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

As shown in FIG. 49, the video display device 100 displays singerinformation, price information, and description information of musicrelated to a main AV content on the additional information area 360.Like this, according to an embodiment, the video display device 100 mayautomatically search and display a music catalog related to a main AVcontent without user's manipulation.

A display example of another catalog list will be described withreference to FIG. 50.

FIG. 50 is a screen when another sub menu of a recommendation main menuin a video display device manufacturer application is selected accordingto an embodiment.

As shown in FIG. 50, the video display device 100 displays priceinformation and description information of products and places relatedto a main AV content on the additional information area 360. Like this,according to an embodiment, the video display device 100 mayautomatically search and display a products or places catalog related toa main AV content without user's manipulation.

Once one item is selected from the catalog list in operation S1809, thevideo display device manufacturer application displays a purchase itemcorresponding to the selected item in operation S1811.

Like this, according to an embodiment, a user may purchase productsthrough simple manipulation.

A post purchase review button and a bookmark button may be disposed ateach item in the catalog list.

Once the post purchase review button is selected, the video displaydevice 100 may display a post use review of a product on the additionalinformation area 360. Additionally, a user may write a post purchasereview.

Additionally, after bookmarking a specific item through the bookmarkbutton, a user may recommend the bookmarked item through a viewer listsub menu of a social main menu.

In this case, the video display device 100 may provide bookmark iteminformation to the ACR server 77 or SNS providing server 75.

Then, the video display device 100 of a user receiving therecommendation obtains content information on the basis of the extractedsection from a main AV content, and may display a recommendation messagesuch as “this product is recommended by a friend U” if the main AVcontent corresponds to an advertisement of the recommended item on thebasis of the obtained content information. Additionally, instead of arecommendation message, a short video may be displayed.

Then, an on/off control of an ACR function will be described accordingto an embodiment with reference to FIGS. 51 and 52.

FIG. 51 is a view of a user interface for controlling an ACR functionaccording to an embodiment.

As shown in FIG. 51, the play controlling unit 113 displays an ACRfunction setting window. A user may activate or deactivate the ACRfunction through the ACR function setting window.

When the ACR function is activated, a user may select an ACR mode. TheACR mode may be set with one of an auto playing application mode, acontent provider application mode, and an video display manufacturerapplication mode.

When the ACR function mode is set in the above manner, the playcontrolling unit 113 stores setting information on a setting informationstorage unit 154.

FIG. 52 is a flowchart illustrating a method of controlling a videodisplay device according to the embodiment.

The enhanced service management unit 130 confirms whether the ACRfunction is turned on in operation S1901.

If the ACR function is deactivated, the enhanced service management unit130 does not perform a content information obtaining procedure any more.

If the ACR function is activated, the enhanced service management unit130 extracts a certain section of a main AV content in operation S1902.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS1903.

The enhanced service management unit 130 synchronizes the reference timeof the main AV content with the reference time of an alternativeadvertisement in operation S1905.

The enhanced service management unit 130 obtains an enhanced service onthe basis of the obtained content information in operation S1907. Theenhanced service management unit 130 obtains enhanced serviceinformation on the basis of the content information, and then obtains anenhanced service on the basis of the enhanced service information.

Then, the enhanced service management unit 130 confirms the ACR mode inoperation S1909.

The enhanced service management unit 130 plays the enhanced serviceaccording to the ACR mode in operation S1911.

In more detail, if the ACR mode is an automatic play application mode,the enhanced service management unit 130 executes an automatic playapplication, and the executed automatic play application plays theobtained enhanced service.

If the ACR mode is the content provider application mode, the enhancedservice management unit 130 executes a content provider applicationcorresponding to the obtained content information in background. Theenhanced service management unit 130 displays a content providerapplication when receiving a user input corresponding to an enhancedservice use. This content provider application may play the obtainedenhanced service. Instead that the content provider application isautomatically executed in background, it may be executed by a userinput.

If the ACR mode is the video display device manufacturer applicationmode, the enhanced service management unit 130 executes its videodisplay device manufacturer application in background. The enhancedservice management unit 130 displays the video display devicemanufacturer application when receiving a user input corresponding to anenhanced service use. This video display device manufacturer applicationmay play the obtained enhanced service. Instead that the video displaydevice manufacturer application is automatically executed in background,it may be executed by a user input.

Next, an advertisement alternative method will be described according toan embodiment with reference to FIGS. 53 and 54.

FIG. 53 is a flowchart illustrating an advertisement alternative methodaccording to an embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2001.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2003.

Then, the enhanced service management unit 130 obtains advertisementalternative information, i.e., enhanced service information, on thebasis of the obtained content information in operation S2005

The enhanced service management unit 130 may receive a query resultincluding both content information and advertisement alternativeinformation of the main AV content. Table 14 illustrates an example ofthe query result.

TABLE 14   <queryresult> <content > < id>Secret Garden</id> < playtimeunit=“mm:ss”>68:00</playtime> < broadcastor>SBS </broadcastor></content> <signature> <starttime unit=“ms”>78000</starttime> <endtimeunit=“ms”>79965</endtime> </signature> <AD> <ad1> <adtype> stream or/andfile </adtype> <category>product</category> <adname>LG LED DTV </adname><starttime unit=“ms”>95000</starttime> <endtimeunit=“ms”>110000</endtime> <adURI>www.lge.co.kr/DTV/lw5700.avi</adURI> <description>LG 3D smart TV </description> </ad1> <ad2> <adtype> streamor/and file </adtype> <category>product</category> <adname>LG smartphone </adname> <starttime unit=“ms”>145000</starttime> <endtimeunit=“ms”>175000</endtime> <adURI>www.lge.co.kr/phone/lu2300.avi</adURI><description>LG smart phone </description> </ad2> ... </AD></queryresult>

In Table 14, a <queryresult> tag is a root tag, and represents a queryreply from the fingerprint server 22. A <content> tag shows entireinformation of a main AV content. A <playtime> tag shows entire playtime of the main AV content. The video display device 100 may calculatethe end time of the main AV content through information in the<playtime> tag. A <broadcastor> tag shows broadcasting stationinformation of the main AV content.

A <signature> tag shows time information of an AV sample that the videodisplay device 100 uses to generate a signature. A <starttime> tag inthe <signature> tag includes the start time of the AV sample used forgenerating a signature, and a <endtime> tag in the <signature> tagincludes the end time of the AV sample used for generating a signature.The video display device 100 may continuously synchronize the play timeof the main AV content through information in the <starttime> tag and<endtime> tag.

A <AD> tag includes all advertisement alternative information related toa content that is currently played in chronological order. The videodisplay device 100 receives and stores an advertisement file or streamin advance or may buffer it through information in the <AD> tagMoreover, the video display device 100 may properly replace anadvertisement at the synchronized time with the main AV content throughthe information in the <AD> tag.

A <ad1> tag includes information on one advertisement used foradvertisement alternative. A <adtype> tag represents a type of anadvertisement, and notifies a receiver that an advertisement is a mediafile or media stream. A <category> tag shows a category item ofadvertisement, and the video display device 100 selects and displays anadvertisement that a user wants if there are several advertisements inthe same advertising time by using the <category> tag. A <adname> tagshows a name of an advertisement.

A <starttime> tag shows a start time of an alternative advertisement,and an <endtime> tag shows an end time of an alternative advertisement.The start time and end time of the alternative advertisement are a timestamp value calculated from the starting of the main AV content and areexpressed in ms. A <adRUL> tag includes position information of analternative advertisement. A <description> tag includes description ofan alternative advertisement.

After the advertisement alternative information is transmitted, only thecontent information shown in Table 15 may be transmitted.

TABLE 15   <queryresult> <content > < id>Secret Garden</id> < playtimeunit=“mm:ss”>68:00</playtime> < broadcastor>SBS </broadcastor></content> <signature> <starttime unit=“ms”>110000</starttime> <endtimeunit=“ms”>115000</endtime> </signature> </queryresult>

The enhanced service management unit 130 accesses an alternativeadvertisement address in the advertisement alternative information toobtain an alternative advertisement file in operation S2007. As shown inTable 14, if the advertisement alternative information includes aplurality of advertisement addresses, the enhanced service managementunit 130 obtains a plurality of alternative advertisement files.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of an alternative advertisementin operation S2009.

If the synchronized play time of the main AV content is identical to thestart time of the alternative advertisement, the enhanced servicemanagement unit 130 starts the alternative advertisement to replace theadvertisement of the main AV content in operation S2011.

FIG. 54 is a flowchart illustrating an advertisement alternative methodaccording to another embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2101.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2103. At this point, the content of the content information correspondsto Table 15.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of an alternative advertisementin operation S2104.

Then, the enhanced service management unit 130 obtains advertisementalternative information, i.e., enhanced service information, on thebasis of the obtained content information in operation S2105.

The enhanced service management unit 130 may receive a query resultincluding both content information and advertisement alternativeinformation of the main AV content. Table 16 illustrates an example ofthe query result.

TABLE 16   <queryresult> <content > < id>Secret Garden</id> < playtimeunit=“mm:ss”>68:00</playtime> < broadcastor>SBS </broadcastor></content> <signature> <starttime unit=“ms”>78000</starttime> <endtimeunit=“ms”>79965</endtime> </signature> <AD> <adtype> stream or/and file</adtype> <category>product</category> <adname>LG LED DTV </adname><starttime unit=“ms”>95000</starttime> <endtimeunit=“ms”>110000</endtime> <adURI>www.lge.co.kr/DTV/lw5700.avi</adURI>< description>LG 3D smart TV </description> </AD> </queryresult>

The enhanced service management unit 130 accesses an alternativeadvertisement address in the advertisement alternative information toobtain an alternative advertisement file in operation S2107.

If the synchronized play time of the main AV content is identical to thestart time of the alternative advertisement, the enhanced servicemanagement unit 130 starts the alternative advertisement to replace theadvertisement of the main AV content in operation S2111.

Next, a scene related information displaying method will be describedaccording to an embodiment with reference to FIGS. 55 and 62.

FIG. 55 is a flowchart illustrating a scene related informationdisplaying method according to an embodiment. FIG. 56 is a userinterface for selecting a scene related information displaying modeaccording to an embodiment.

When a user selects the scene related information displaying mode from aplurality of menus by a user in operation S2201, the enhanced servicemanagement unit 130 displays a screen for selecting at least one of aplurality of scene related information display modes on the display unit121 in operation S2203.

As shown in FIG. 56, the plurality of scene related information displaymodes may include an immediate display mode, a user selection displayingmode, an integrated display mode, and a scene related informationdisregarding mode.

When one of the plurality of scene related information displaying modesis selected in operation S2205, the enhanced service management unit 130displays a screen for selecting at least one of a plurality of scenerelated information category items on the display unit 21 in operationS2207.

As shown in FIG. 56, the plurality of scene related information categoryitems may include various items such as filming places, cuisines, andrestaurants.

When at least one of the plurality of scene related information categoryitems is selected in operation S2209, the enhanced service managementunit 130 stores the selected scene related information displaying modeand selected scene related information category item on the settinginformation storage unit 157 in operation S2211.

FIG. 57 is flowchart illustrating a scene related information displayingmethod according to an embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2301.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2303.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of scene related information,i.e., enhanced service information, in operation S2304.

Then, the enhanced service management unit 130 obtains the scene relatedinformation, i.e., enhanced service information, on the basis of theobtained content information in operation S2305.

The enhanced service management unit 130 may receive a query resultincluding both content information and scene related information of themain AV content. Table 17 illustrates an example of the query result.

TABLE 17   <queryresult> <content > < id>Secret Garden</id> < playtimeunit=“mm:ss”>68:00</playtime> < broadcastor>SBS </broadcastor></content> <signature> <starttime unit=“ms”>78000</starttime> <endtimeunit=“ms”>79965</endtime> </signature> <sceneRelatedInfo><category>food</category> <infoname>food name 1 </infoname> <starttimeunit=“ms”>95000</starttime> <endtime unit=“ms”>110000</endtime> <iconURI>www.food.co.kr/food/name1.jpg</iconURI><infoURI>www.food.co.kr/food/name1.html</infoURI> < description>foodname1's recipe </description> </sceneRelatedInfo> </queryresult>

The enhanced service management unit 130 accesses a scene relatedinformation address in the query result to obtain the scene relatedinformation in operation S2307.

If the synchronized play time of the main AV content is identical to thestart time of the scene related information in operation S2308, theenhanced service management unit 130 processes the scene relatedinformation according to a scene related information displaying mode andscene related information category item stored in the settinginformation storage unit 157.

At this point, if the selected scene related information displaying modeis an immediate display mode and a category of the obtained scenerelated information corresponds to a scene related information categoryitem selected related to the selected scene related informationdisplaying mode in operation S2309, the enhanced service management unit130 plays the obtained scene related information regardless of user'smanipulation in operation S2310.

At this point, if the selected scene related information displaying modeis a scene related information disregarding mode and a category of theobtained scene related information corresponds to a scene relatedinformation category item selected related to the selected scene relatedinformation displaying mode in operation S2311, the enhanced servicemanagement unit 130 may disregard the obtained scene related informationin operation S2312. That is, the enhanced service management 130 may notdisplay an indicator for scene related information and may not executescene related information. At this point, if the selected scene relatedinformation displaying mode is a scene related information disregardingmode, the enhanced service management unit 130 may disregard theobtained scene related information regardless of the selected scenerelated information category item.

If the selected scene related information displaying mode is a userselection displaying mode and a category item of the obtained scenerelated information corresponds to a scene related information categoryitem selected related to the selected scene related informationdisplaying mode in operation S2313, the enhanced service management unit130 may display an indicator for notifying that there is the obtainedscene related information on the display unit 121, and may play theobtained scene related information in correspondence to an enhancedservice display user input in the play device 120. This will bedescribed with reference to FIGS. 58 and 59.

FIG. 58 is a flowchart illustrating a user selection displaying methodof scene related information according to an embodiment. FIG. 59 isconceptual diagram illustrating a user selection displaying method ofscene related information according to an embodiment.

Once it reaches the start time of an application 1 corresponding toscene related information, the video display device 100 displays atleast one indicator on the display unit 121 in operation S2315. As shownin FIG. 59, at least one indicator may include an immediate viewindicator, a bookmarking indicator, and a close indicator.

When receiving a user input corresponding to the immediate viewindicator in operation S2316, the video display device 100 plays theobtained scene related information in the play device 120 in operationS2317.

When receiving a user input corresponding to the close indicator inoperation S2319, the video display device 100 closes the displayedindicator and terminates the obtained scene related information inoperation S2321.

When receiving a user input corresponding to the bookmarking indicatorin operation S2323, the video display device 100 stores the bookmark ofthe obtained scene related information in the storage unit 155 inoperation S2325. At this point, the bookmark includes a bookmarkidentifier, a scene related information address, scene relatedinformation description, available time of scene related information,and a capture image of a main AV content corresponding to the scenerelated information.

Later, when the video display device 100 receives a user inputcorresponding to bookmark display or it reaches a predetermined bookmarkdisplay time in operation 2327, the video display device 100 may displayat least one bookmark on the display unit 121 in operation S2329. Thepredetermined set bookmark display time may be the end time of a main AVcontent. A plurality of bookmarks may be displayed by using a categoryitem and available time as a classifier.

When a user clicks the scene related information address or descriptionin operation S2331, the video display device 100 plays the clicked scenerelated information in operation S2333.

Additionally, when a user clicks a capture image of a main AV contentcorresponding to the scene related information in operation S2335, thevideo display device 100 plays the main AV content at the pointcorresponding to the capture image in operation S2337. At this point,the video display device 100 may receive the main AV content in a streamformat and may play it.

If the selected scene related information displaying mode is anintegrated display mode and a category item of the obtained scenerelated information corresponds to a scene related information categoryitem selected related to the selected scene related informationdisplaying mode in operation S2314, the enhanced service management unit130 stores the bookmark of the scene related information obtainedregardless of user's manipulation in the bookmark storage unit 155. Thiswill be described with reference to FIGS. 60 and 61.

FIG. 60 is a flowchart illustrating a method of displaying integratedscene related information according to an embodiment. FIG. 61 isconceptual diagram illustrating a method of displaying integrated scenerelated information according to an embodiment.

If the selected scene related information displaying mode is anintegrated display mode and a category item of the obtained scenerelated information corresponds to a scene related information categoryitem selected related to the selected scene related informationdisplaying mode, the enhanced service management unit 100 stores thebookmark of the scene related information obtained regardless of user'smanipulation in the bookmark storage unit 155 in operation S2339,without displaying an indicator. At this point, the bookmark includes abookmark identifier, a scene related information address, scene relatedinformation description, available time of scene related information,and a capture image of a main AV content corresponding to the scenerelated information.

Later, when the video display device 100 receives a user inputcorresponding to bookmark display or it reaches a predetermined bookmarkdisplay time in operation 2341, the video display device 100 may displayat least one bookmark on the display unit 121 in operation S2343. Thepredetermined set bookmark display time may be the end time of a main AVcontent. As shown in FIG. 61, a plurality of bookmarks may be displayedby using a category item and available time as a classifier.

When a user clicks the scene related information address or descriptionin operation S2345, the video display device 100 plays the clicked scenerelated information in operation S2347.

Additionally, when a user clicks a capture image of a main AV contentcorresponding to the scene related information in operation S2349, thevideo display device 100 plays the main AV content at the pointcorresponding to the capture image in operation S2351. At this point,the video display device 100 may receive the main AV content in a streamformat and may play it.

If a method of displaying the integrated scene related information isused as shown in FIG. 62, sponsors of the main AV content may beintegrated and displayed.

FIG. 62 illustrates an integrated screen of sponsor advertisements of amain AV content.

As shown in FIG. 62, when the video display device 100 receives a userinput corresponding to bookmark display or it reaches a predeterminedbookmark display time, the video display device 100 may display anintegrated screen of sponsor advertisements of the main AV content. Atthis point, sponsor advertisement items of the main AV content aredisplayed in an icon form, and once an icon is clicked, the videodisplay device 100 may display a service page corresponding to theclicked icon.

Next, referring to FIGS. 63 and 67, a method of controlling the videodisplay device 100 according to rating information will be describedaccording to an embodiment.

FIG. 63 is a flowchart illustrating a method of storing rating settinginformation according to an embodiment. FIG. 64 is a user interface forstoring rating setting information according to an embodiment.

When a user selects a rating setting menu from a plurality of menus inoperation S2401, the management unit 130 displays a screen for passwordinput on the display unit 121 in operation S2403.

If an accurate password is inputted in operation S2405, the enhancedservice management unit 130 displays a rating setting screen on thedisplay unit 121 in operation S2407. As shown in FIG. 64, a plurality ofrating setting items may be inputted through a rating setting screen.Each rating setting item may include a rating applying time and a ratingvalue. The rating value may include available all ages, available over 7years, available over 12 years, and available over 19 years.

Once rating setting information is inputted through the rating settingscreen in operation S2409, the enhanced service management unit 130stores the inputted rating setting information in the settinginformation storage unit 157 in operation S2441.

FIG. 65 is flowchart illustrating a method of playing a rating basedmain AV content according to an embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2501.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2503. Table 18 illustrates an example of the obtained contentinformation.

TABLE 18   <queryresult> <content> <id>Secret Garden</id> <playtimeunit=“mm:ss”>68:00</playtime> <broadcastor>SBS </broadcastor> <ratingcountry=“Korea” value=“12 

 ” type=“ 

 ”> </content> <signature> <starttime unit=“ms”>78000</starttime><endtime unit=“ms”>79965</endtime> </signature> </queryresult>

As shown in Table 18, the rating of the current main AV contentcorresponds to available over 12 years.

The enhanced service management unit 130 determines whether the main AVcontent rating information in the obtained content information satisfiesthe set main AV rating information in operation S2505.

If the main AV content rating information in the obtained contentinformation does not satisfy the set main AV rating information, theenhanced service management unit 130 obtains an enhanced service forpreventing a main AV content from being played such as mosaic, picture,advertisement, and password input screen and then plays the obtainedenhanced service in operation S2507.

If an accurate password is inputted through the password input screen inoperation S2509, the enhanced service management unit 130 cancels theenhanced service for preventing the main AV content from being played inoperation S2507.

The enhanced service management unit 130 records an event that does notsatisfy the set main AV rating information and an event that cancels theenhanced service for preventing the main AV content from being played inoperation S2513.

Then, the enhanced service management unit 130 reports the recordedevent information to the predetermined e-mail according to apredetermined reporting time or a user request in operation S2515.

FIG. 66 is flowchart illustrating a method of playing a rating basedmain AV content according to another embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2601.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2603. Table 19 illustrates an example of the obtained contentinformation.

TABLE 19 <queryresult> <content > <id>Secret Garden</id> <playtime unit=“mm:ss” >68:00</playtime> <broadcastor>SBS </broadcastor> <ratingcountry= “Korea” type= “ 

 ” > </content> <signature> <starttime unit= “ms” >78000</starttime><endtime unit= “ms” >79965</endtime> </signature> <sceneRating> <scene1startTime= “30” endTime= “100” value= “12 

 ” > <scene2 startTime= “150” endTime= “300” value= “19 

 ” > <scene3 startTime= “700” endTime= “1000” value= “19 

 ” > <scene4 startTime= “3000” endTime= “10000” value= “12 

 ” > ... </sceneRating> </queryresult>

As shown in Table 19, the content information includes ratinginformation having a plurality of rating information items correspondingto a plurality of time slots, respectively. Each rating information itemhas a start time, an end time, and a rating value.

In Table 19, a rating value of a time slot between 30 sec and 100 sec ofa current main AV content corresponds to available over 12 years; arating value of a time slot between 15 sec and 300 sec of a current mainAV content corresponds to available over 19 years; a rating value of atime slot between 700 sec and 1000 sec of a current main AV contentcorresponds to available over 19 years; and a rating value of a timeslot between 3000 sec and 10000 sec of a current main AV contentcorresponds to available over 12 years.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the time of the rating information in operationS2604.

The enhanced service management unit 130 determines whether the ratinginformation at the timing of when the main AV content is playedsatisfies the set main AV rating information on the basis of the main AVcontent rating information in the obtained content information inoperation S2605.

If the rating information at the timing of when the main AV content isplayed does not satisfy the set main AV rating information, the enhancedservice management unit 130 obtains an enhanced service for preventing amain AV content from being played such as mosaic, picture,advertisement, and password input screen and then plays the obtainedenhanced service in operation S2607.

If an accurate password is inputted through the password input screen inoperation S2609, the enhanced service management unit 130 cancels theenhanced service for preventing the main AV content from being played inoperation S2611.

The enhanced service management unit 130 records an event that does notsatisfy the set main AV rating information and an event that cancels theenhanced service for preventing the main AV content from being played inoperation S2613.

Then, the enhanced service management unit 130 reports the recordedevent information to the predetermined e-mail according to apredetermined reporting time or a user request in operation S2615.

FIG. 67 is flowchart illustrating a method of playing a rating basedmain AV content according to another embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2701.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2703.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of a deletion section AVcontent, i.e., enhanced service, in operation S2704.

Then, the enhanced service management unit 130 obtains the enhancedservice information on the basis of the obtained content information inoperation S2705.

The enhanced service management unit 130 may receive a query resultincluding both content information and enhanced service information ofthe main AV content. Table 20 illustrates an example of the queryresult.

Table 20 illustrates an example of the obtained content information.

TABLE 20   <queryresult> <content > <id>Secret Garden</id> <playtimeunit= “mm:ss” >68:00</playtime> <broadcastor>SBS </broadcastor> <ratingcountry= “Korea” type= “ 

 ” > </content> <signature> <starttime unit= “ms” >78000</starttime><endtime unit= “ms” >79965</endtime> </signature> <deletedScene> <scene1startTime= “30:00” endTime= “35:00” rating= “19 

 ” > <avlocation> www.movie.com/part/12345.avi</avlocation> </scene1><scene2 startTime= “55:30” endTime= “61:40” rating= “19 

 ” > <avlocation> www.movie.com/part/12346.avi</avlocation> </scene2></deletedScene> </queryresult>

As shown in Table 20, the content information includes ratinginformation having a plurality of rating information items correspondingto a plurality of time slots, respectively. Each rating information itemhas a start time, an end time, a rating value, and a deletion AV contentaddress.

As shown in Table 20, a rating value of a time slot between 30 min and35 min of a current main AV content corresponds to available over 19years, and an AV content of a deleted corresponding time slot isobtained from a deletion AV content address“www.movie.com/part/12345.avi”. A rating value of a time slot between 55min 30 sec and 61 min 40 sec of a current main AV content corresponds toavailable over 19 years, and an AV content of a deleted correspondingtime slot is obtained from a deletion AV content address“www.movie.com/part/12346.avi”.

The enhanced service management unit 130 accesses an address of an AVcontent of a deletion section to obtain the deleted AV content inoperation S2707.

The enhanced service management unit 130 determines whether the ratinginformation of the AV content in the deletion section satisfies the setmain AV rating information in operation S2709.

If the rating information of the AV content in the deletion sectionsatisfies the set main AV rating information and the synchronized playtime of the main AV content is identical to the start time of the AVcontent in the deletion section, the enhanced service management unit130 stops playing the main AV content, stores the received main AVcontent, and plays the AV content in the deletion time slot in operationS2711.

If the synchronized play time of the main AV content is identical to theend time of the AV content in the deletion section, the enhanced servicemanagement unit 130 plays the stored main AV content from the time ofwhen the main AV content playback stops to provide a continuous AVcontent in operation S2712.

The enhanced service management unit 130 records an event that plays amain AV content in the deletion time slot in operation S2713.

Then, the enhanced service management unit 130 reports the recordedevent information to the predetermined e-mail according to apredetermined reporting time or a user request in operation S2715.

Next, a usage information reporting method will be described accordingto an embodiment with reference to FIGS. 68 and 76.

The video display device 100 may include an operation for registeringitself to a usage information measuring server to report usageinformation, an operation for obtaining usage information, and anoperation for reporting and using usage information.

FIG. 68 is flowchart illustrating a method of registering to a usageinformation measuring server according to an embodiment.

As shown in FIG. 68, the video display device 100 communicates with theusage information measuring server 80. The usage information measuringserver 80 may be one module in the content providing server 10, thecontent recognizing service providing server 20, the multi channel videodistributing server 30, the enhanced service information providingserver 40, the plurality of enhanced service providing servers 50, thebroadcast receiving device 60, or the video display device 100, or maybe an additional server inside or outside a home.

First, the video display device 100 collects user information inoperation S2801. The video display device 100 receives and collectsaccount information, family member information, preferred genreinformation, and usage information range from a user, and then, storesthe collected user information in the user information storage unit 156.The video display device 100 may collect video display deviceinformation from the user information storage unit 156.

Then, the video display device 100 generates a registration requestmessage including the collected user information in operation S2803. Theregistration request message may include an XML type document as shownin FIG. 69.

FIG. 69 is a view illustrating an XML schema of user informationaccording to an embodiment.

As shown in FIG. 69, a UserProfile element containing user informationhas id, password, and terminalID attributes and Region, Demographic,Favorite, SNS, and Terminal elements.

The id and password attributes include account information for the usageinformation measuring server 80. The terminalID attribute includes anidentifier of the video display device 100.

The Region element includes Country, City, Address, and Zip elements.The Country element includes country information; the City elementincludes city information; the Address element includes addressinformation; and the Zip element includes zip code. Information in theRegion element allows different advertisement and product introductionaccording to a region.

The Demographic element includes Age, Sex, Religion, Income, and FamilyType elements. The Age element includes age information; the Sex elementincludes sex information; and the Religion element includes religioninformation. The Income element includes household income information,and the Family Type element includes family member type information.Information in the Demographic element allows customized serviceaccording to a family member type.

The Favorite element includes MainCategory and SubCategory elements. TheMainCategory element includes a main preferred genre, and theSubCategory element includes a sub preferred genre.

The SNS element includes Facebook and Twitter elements. The Facebookelement includes Facebook account information, and the Twitter elementincludes Twitter account information.

The Terminal element includes Type, Manufacturer, FirmwareVer,Resolution, Model, OS, Browser, Storage, and Network elements. The typeelement includes video display device type information; the Manufacturerelement includes video display manufacturer information; and theFirmwareVer element includes firmware version information of a videodisplay device. The Resolution element includes resolution informationand a video display device; the Model element includes model informationof a video display device; and the OS element includes OS information ofa video display device. The Browser element includes browser informationand a video display device; the Storage element includes storageinformation of a video display device; and the Network element includesnetwork information of a video display device.

Moreover, the registration request message may further include a usageinformation range. The usage information range may be set in eachvirtual channel. Additionally, the usage information range may be setover an entire physical channel.

The video display device 100 transmits the generated registrationrequest message to the usage information measuring server 80 to requesta registration in operation S2805.

FIG. 70 is flowchart illustrating a method of obtaining, reporting, andusing usage information according to an embodiment.

First, the enhanced service management unit 130 extracts a certainsection of a main AV content in operation S2901.

The enhanced service management unit 130 obtains content information ofthe main AV content on the basis of the extracted section in operationS2903. In more detail, the enhanced service management unit 130 decodesinformation encoded with invisible watermark in the extracted section toobtain content information. Additionally, the enhanced servicemanagement unit 130 may extract feature information in the extractedsection, and obtain the content information of the main AV content fromthe fingerprint server 22 or the content information storage unit 151 onthe basis of the extracted feature information.

The enhanced service management unit 130 synchronizes the play time ofthe main AV content with the start time of enhanced service informationin operation S2904.

Then, the enhanced service management unit 130 obtains the enhancedservice information on the basis of the obtained content information inoperation S2905.

The enhanced service management unit 130 obtains an enhanced service onthe basis of the obtained content information in operation S2907.

The enhanced service management unit 130 collects main AV contentwatching information in a usage information range on the basis ofcontinuously obtained content information in operation S2909, and storesthe collected main AV content watching information in the usageinformation storage unit 157. The main AV content watching informationmay include a main AV content identifier and main AV watching timeinformation. The main AV content watching time information may includethe watching start time of and watching end time of the main AV content.

The enhanced service management unit 130 collects enhanced service usageinformation in a usage information range on the basis of a user inputcorresponding to the execution, manipulation, and ending of the obtainedenhanced service in operation S2911, and stores the collected enhancedservice usage information in the usage information storage unit 157. Theenhanced service usage information may include an enhanced serviceidentifier and enhanced service usage time information. The enhancedservice usage time information may include the usage start time andusage end time of the enhanced service.

The enhanced service management unit 130 obtains the address andreporting period of the usage information measuring server 80 inoperation S2913. The address and reporting period of the usageinformation measuring server 80 may be predetermined. The enhancedservice management unit 130 obtains the address and reporting period ofthe usage information measuring server 80 from the content informationor enhanced service information.

In the other hand, the enhanced service management unit 130 obtainscontent information and enhanced service information on the basis of theextracted main AV content section, and obtains ATSC 2.0 metadata fromthe obtained information. The enhanced service management unit 130extracts a usage reporting location table such as Table 21 or Table 22from a service signaling channel of an ATSC 2.0 metadata service, andobtains a reporting address and reporting period from the table. Theusage reporting location table may have a low signaling period such as a1 min period, and this may reduce bandwidth usage.

TABLE 21 Syntax No. Bits Format usage_reporting_location_table_section() {   table_id 8 OxTBD   section_syntax_indicator 1 ‘0’  private_indicator 1 ‘1’   reserved 2 ‘11’   section_length 12 uimsbf  source_id 16 uimsbf   reserved 2 ‘11’   version_number 5 uimsbf  current_next_indicator 1 ‘1’   section_number 8 uimsbf  last_section_number 8 uimsbf   num_usage_reporting_location_in_section8 uimsbf   URL_count 5 uimsbf   for (i=0; 1<URL_count; i++) { uimsbf    type_of_URL 3 uimsbf     reserved 5 uimsbf     URL_length 8 uimsbf    URL( ) var uimsbf     reporting interval 5 uimsbf   } }

As shown in Table 21, the usage reporting location table may signal aplurality of reporting addresses and reporting periods. The plurality ofreporting periods correspond to the plurality of reporting addresses,respectively.

TABLE 22 Syntax No. Bits Format usage_reporting_location_table_section() {   table_id 8 0xTBD   section_syntax_indicator 1 ‘0’  private_indicator 1 ‘1’   reserved 2 ‘11’   section_length 12 uimsbf  source_id 16 uimsbf   reserved 2 ‘11’   version_number 5 uimsbf  current_next_indicator 1 ‘1’   section_number 8 uimsbf  last_section_number 8 uimsbf   num_usage_reporting_location_in_section8 uimsbf    reporting_interval 5 uimsbf    URL_count 5 uimsbf   for(i=0; i<URL_count; i++) { uimsbf      type_of_URL 3 uimsbf      reserved5 uimsbf     URL_length 8 uimsbf     URL( ) var uimsbf   } }

As shown in Table 22, the usage reporting location table may signal aplurality of reporting addresses and one common reporting period.

The enhanced service management unit 130 generates reporting informationincluding at least one main AV content watching information item and atleast one enhanced service usage information item in operation S2915.

Examples of a main AV content watching information item and an enhancedservice usage information item will be described with reference to FIGS.71 and 72.

FIG. 71 is an XML schema of a main AV content watching information itemaccording to an embodiment.

As shown in FIG. 71, an AudienceMeasurement element corresponding to themain AV content watching information item includes an id attribute, apassword attribute, and a terminalID attribute, and a ContentInfoelement and a TimeInfo element.

The id and password attributes include account information for the usageinformation measuring server 80. The terminalID attribute includes anidentifier of the video display device 100.

The ContentInfo element includes an Id element, a Name element, aChallelName element, a Country element, a Genre element, and an HDelement.

The ID element includes a main AV content identifier. Especially, as themain AV content identifier, the Id element may include a transportstream identifier Transport Stream ID, a source identifier source_id, anATSC content identifier, or a global service identifier.

The Name element includes a main AV content name; the ChannelNameelement includes a channel name of a broadcasted main AV content; theCountry element includes country information of a broadcasted main AVcontent; and the Genre element includes genre information of a main AVcontent.

The TimeInfo element includes a TotalWatchingTime element and a Slotelement.

The TotalWatchingTime element includes a main AV content watching timeinformation.

An example of the enhanced service usage information item will bedescribed with reference to FIG. 72.

FIG. 72 is an XML schema of an enhanced service usage information itemaccording to an embodiment.

As shown in FIG. 72, a UsageMonitor element corresponding to theenhanced service usage information item includes an id attribute, apassword attribute, a terminalID attribute, and a ServiceInfo element.

The ServiceInfo element includes a bound attribute and a simplyDisplayattribute, and an Id element, a Name element, a Category element, aPurchase element, and a UserInteraction element.

The bound attribute is set with bound or unbound. The bound means thatan enhanced service is linked with a main AV content, and the unboundmeans that an enhanced service is not linked with a main AV content.

The simplyDisplay attribute is set with a simple information display oruser interaction.

The Id element includes an enhanced service identifier; the Name elementincludes an enhanced service name; and the category element includes anenhanced service category.

The Purchase element includes a ProductName element, a MakerNameelement, an AdditionalInfo element, a Price element, and a ContentIDelement. The ProductName element includes an enhanced service productname; the MakerName element includes a manufacturer name of an enhancedservice product; the AdditionalInfo element includes additionalinformation on an enhanced service; the Price element includes priceinformation of an enhanced service product; and a ContentID elementincludes a content identifier of an enhanced service product.

The UserInteraction element includes a Time element and a Level element.

The Time element includes a From element and a To element. The Fromelement includes an enhanced service usage start time and the To elementincludes an enhanced service usage end time.

The Level element includes information on no interaction, triggerselection, advertisement content watching, product purchase, and fullinteraction.

Moreover, as shown in Table 23, contents of the ID element and timeelement may vary according to a category of an enhanced service.

TABLE 23 Category Field Name Descriptions Web TSID TSID (TransportStream ID) in Bookmark currently watching Set_time Time for setting WebBookmark Follow_time Time for again accessing Web Bookmark and usingenhanced service NRT TSID TSID received/receiving NRT content serviceSource_id source_id of NRT content Service_id Serviceid of NRT contentContent_linkage content_linkage of NRT content Action Downloadstart/Download end/ Subscribe/Unsubscribe/launching/ TerminatingStat_time Action start time for NRT content End_time Action end time forNRT content (there may be no meaning according to Action)

As shown in Table 23, if the enhanced service corresponds to a webbookmark service, the Id element includes TSID, and the Time elementincludes the time for setting a Web Bookmark and the time of when anenhanced service used after accessing the Web Bookmark again.

If the enhanced service corresponds to a non-real-time (NRT) service,the Id element includes TSID, source_id, service_id, andcontent_linkage, and the Time element includes Action, start_time, andend_time.

The enhanced service management unit 130 logs in the usage informationmeasuring server 80 through account information in operation S2917. Thisoperation may be omitted if necessary.

The enhanced service management unit 130 transmits reporting informationto the usage information measuring server 80 corresponding to areporting address in a reporting period in operation S2919. At thispoint, the enhanced service management unit 130 transmits the reportinginformation through HTTP request and terminates the reporting whenreceiving HTTP reply. Additionally, the enhanced service 130 transmitsan HTTP request including a unique identifier such as the identifier ofthe video display device 100 to the usage information measuring server80, and transmits reporting information after receiving a replyincluding a reporting request from the usage information measuringserver 80.

The usage information measuring server 80 provides the accumulated usageinformation to the content providing server 10 in operation S2921, inorder to allow the content providing server 10 to use it. Especially,the content providing server 10 provides customizing content and targetadvertisement, and charges for a bill on the basis of the accumulatedusage information.

Moreover, the usage information measuring server 80 may recognizestatistical and behavioral characteristics of a user on the basis ofviewer information and reported usage information in operation S2923.

The enhanced service management unit 130 obtains various enhancedservices on the basis of the obtained user characteristics in operationS2925. A user characteristic based enhanced service will be describedwith reference to FIGS. 73 to 76. If the usage information measuringserver 80 is one module in the video display device 100, the enhancedservice management unit 130 may obtain an enhanced service from theenhanced service storage unit 152. If the usage information measuringserver 80 is one module in the content providing server 10, the contentrecognizing service providing server 20, the multi channel videodistributing server 30, the enhanced service information providingserver 40, the plurality of enhanced service providing servers 50, orthe broadcast receiving device 60, it may obtain an enhanced servicefrom a corresponding device.

FIGS. 73 to 76 are conceptual diagrams of a user characteristic basedenhanced service according to an embodiment.

As shown in FIG. 73, a user characteristic is that a user enjoyswatching a specific AV content. In this case, the enhanced servicemanagement unit 130 may obtain an enhanced service that recommends achange to a channel in which a specific AV content is broadcasted. Theenhanced service management unit 130 displays the obtained enhancedservice, and receives and plays an AV content of a corresponding channelwhen receiving a user input for channel change confirmation.

As shown in FIG. 74, a user characteristic is that a user enjoyswatching a specific genre. In this case, the enhanced service managementunit 130 may obtain an enhanced service that recommends a current AVcontent or enhanced service corresponding to a user's favorite genre.The enhanced service management unit 130 displays the obtained enhancedservice, and receives and plays a corresponding AV content or enhancedservice when receiving a user input for watching confirmation.

As shown in FIG. 75, a user characteristic is that a user enjoyswatching TV at a specific time. In this case, the enhanced servicemanagement unit 130 may receive a target advertisement for the specifictime that a user usually watches TV, and may play the received targetadvertisement.

As shown in FIG. 76, users having similar user characteristic may begrouped. On the basis of determining which user of the video displaydevice 100 is included which group, contents or enhanced services thatusers of a corresponding group prefer most may be recommend. Theenhanced service management unit 130 obtains an enhanced service, whichrecommends an AV content that a user having similar user characteristicselects or watches currently. Additionally, the enhanced servicemanagement unit 130 obtains an enhanced service, which recommends an AVcontent that a similar user on a social network selects or watchescurrently.

Next, a structure of a video display device according to variousembodiments will be described with reference to FIGS. 77 and 78.

FIG. 77 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

As shown in FIG. 77, a tuner 501 extracts a symbol from an 8-VSB RFsignal transmitted through an air channel.

An 8-VSB demodulator 503 demodulates the 8-VSB symbol that the tuner 501extracts and restores meaningful digital data.

A VSB decoder 505 decodes the digital data that the 8-VSB demodulator503 to restore an ATSC main service and ATSC M/H service.

An MPEG-2 TP Demux 507 filters a Transport Packet that the video displaydevice 100 is to process from an MPEG-2 Transport Packet transmittedthrough an 8-VSB signal or an MPEG-2 Transport Packet stored in a PVRStorage to relay the filtered Transport Packet into a processing module.

A PES decoder 539 buffers and restores a Packetized Elementary Streamtransmitted through an MPEG-2 Transport Stream.

A PSI/PSIP decoder 541 buffers and analyzes PSI/PSIP Section Datatransmitted through an MPEG-2 Transport Stream. The analyzed PSI/PSIPdata are collected by a Service Manager (not shown), and then, is storedin DB in a form of Service Map and Guide data.

A DSMCC Section Buffer/Handler 511 buffers and processes DSMCC SectionData for file transmission through MPEG-2 TP and IP Datagramencapsulation.

An IP/UDP Datagram Buffer/Header Parser 513 buffers and restores IPDatagram, which is encapsulated through DSMCC Addressable section andtransmitted through MPEG-2 TP to analyze the Header of each Datagram.Additionally, an IP/UDP Datagram Buffer/Header Parser 513 buffers andrestores UDP Datagram transmitted through IP Datagram, and then analyzesand processes the restored UDP Header.

A Stream component handler 557 may include ES Buffer/Handler, PCRHandler, STC module, Descrambler, CA Stream Buffer/Handler, and ServiceSignaling Section Buffer/Handler.

The ES Buffer/Handler buffers and restores an Elementary Stream such asVideo and Audio data transmitted in a PES form to deliver it to a properA/V Decoder.

The PCR Handler processes Program Clock Reference (PCR) Data used forTime synchronization of Audio and Video Stream.

The STC module corrects Clock values of the A/V decoders by using aReference Clock value received through PCR Handler to perform TimeSynchronization.

When scrambling is applied to the received IP Datagram, the Descramblerrestores data of Payload by using Encryption key delivered from the CAStream Handler.

The CA Stream Buffer/Handler buffers and processes Data such as Keyvalues for Descrambling of EMM and ECM, which are transmitted for aConditional Access function through MPEG-2 TS or IP Stream. An output ofthe CA Stream Buffer/Handler is delivered to the Descrambler, and then,the descrambler descrambles MPEG-2 TP or IP Datagram, which carriers A/VData and File Data.

The Service Signaling Section Buffer/Handler buffers, restores, andanalyzes NRT Service Signaling Channel Section Data transmitted in aform of IP Datagram. The Service Manager (not shown) collects theanalyzed NRT Service Signaling Channel Section data and stores them inDB in a form of Service Map and Guide data.

The A/V Decoder 561 decodes the Audio/Video data received through an ESHandler to present them to a user.

An MPEG-2 Service Demux (not shown) may include an MPEG-2 TPBuffer/Parser, a Descrambler, and a PVR Storage module.

An MPEG-2 TP Buffer/Parser (not shown) buffers and restores an MPEG-2Transport Packet transmitted through an 8-VSB signal, and also detectsand processes a Transport Packet Header.

The Descrambler restores the data of Payload by using an Encryption key,which is delivered from the CA Stream Handler, on the Scramble appliedPacket payload in the MPEG-2 TP.

The PVR Storage module stores an MPEG-2 TP received through an 8-VSBsignal at the user's request and outputs an MPEG-2 TP at the user'srequest. The PVR storage module may be controlled by the PVR manager(not shown).

The File Handler 551 may include an ALC/LCT Buffer/Parser, an FDTHandler, an XML Parser, a File Reconstruction Buffer, a Decompressor, aFile Decoder, and a File Storage.

The ALC/LCT Buffer/Parser buffers and restores ALC/LCT data transmittedthrough a UDP/IP Stream, and analyzes a Header and Header extension ofALC/LCT. The ALC/LCT Buffer/Parser may be controlled by an NRT ServiceManager (not shown).

The FDT Handler analyzes and processes a File Description Table of FLUTEprotocol transmitted through an ALC/LCT session. The FDT Handler may becontrolled by an NRT Service Manager (not shown).

The XML Parser analyzes an XML Document transmitted through an ALC/LCTsession, and then, delivers the analyzed data to a proper module such asan FDT Handler and an SG Handler.

The File Reconstruction Buffer restores a file transmitted through anALC/LCT, FLUTE session.

If a file transmitted through an ALC/LCT and FLUTE session iscompressed, the Decompressor performs a process to decompress the file.

The File Decoder decodes a file restored in the File ReconstructionBuffer, a file decompressed in the decompressor, or a film extractedfrom the File Storage.

The File Storage stores or extracts a restored file if necessary.

The M/W Engine (not shown) processes data such as a file, which is notan A/V Stream transmitted through DSMCC Section and IP Datagram. The M/WEngine delivers the processed data to a Presentation Manager module.

The SG Handler (not shown) collects and analyzes Service Guide datatransmitted in an XML Document form, and then, delivers them to the EPGManager.

The Service Manager (not shown) collects and analyzes PSI/PSIP Datatransmitted through an MPEG-2 Transport Stream and Service SignalingSection Data transmitted through an IP Stream, so as to produce aService Map. The Service Manager (not shown) stores the produced servicemap in a Service Map & Guide Database, and controls an access to aService that a user wants. The Service Manager is controlled by theOperation Controller (not shown), and controls the Tuner 501, the MPEG-2TP Demux 507, and the IP Datagram Buffer/Handler 513.

The NRT Service Manager (not shown) performs an overall management onthe NRT service transmitted in an object/file form through a FLUTEsession. The NRT Service Manager (not shown) may control the FDT Handlerand File Storage.

The Application Manager (not shown) performs overall management onApplication data transmitted in a form of object and file.

The UI Manager (not shown) delivers a user input to an OperationController through a User Interface, and starts a process for a servicethat a user requests.

The Operation Controller (not shown) processes a command of a user,which is received through a UI Manager, and allows a Manager of anecessary module to perform a corresponding action.

The Fingerprint Extractor 565 extracts fingerprint feature informationfrom an AV stream.

The Fingerprint Comparator 567 compares the feature informationextracted by the Fingerprint Extractor with a Reference fingerprint tofind an identical content. The Fingerprint Comparator 567 may use aReference fingerprint DB stored in local and may query a Fingerprintquery server on the internet to receive a result. The matched resultdata obtained by a comparison result may be delivered to Application andused.

As an ACR function managing module or an application module providing anenhanced service on the basis of ACR, the Application 569 identifies abroadcast content in watching to provide an enhanced service related toit.

FIG. 78 is a block diagram illustrating a structure of a watermark basedvideo display device according to another embodiment.

Although the watermark based video display device of FIG. 78 is similarto the fingerprint based video display device of FIG. 77, thefingerprint based video display device does not includes the FingerprintExtractor 565 and the Fingerprint Comparator 567, but further includesthe Watermark Extractor 566.

The Watermark Extractor 566 extracts data inserted in a watermark formfrom an Audio/Video stream. The extracted data may be delivered to anApplication and may be used.

According to embodiments, services led by a broadcasting station areavailable. Especially, the embodiments provide alternative advertisingservices, scene associated information providing services, rating-basedcontent services, and user attribute-based services.

Additionally, according to embodiments, even when receiving AV contentshaving no time information or AV contents having a different referencetime than enhanced service, a video display device may synchronize thetime-sensitive enhanced service with the AV contents.

Moreover, according to embodiments, information search, chatting, newssearch, digital media download, product catalog download, and productpurchase, which are related to main AV contents, are available withoutinconvenient text input using a remote controller.

Additionally, according to embodiments, enhanced services exceeding thebandwidth limitation of a wireless channel become available.

Moreover, the above methods of the present invention can also beembodied as computer readable codes on a computer readable recordingmedium. Examples of the computer readable recording medium includeread-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetictapes, floppy disks, optical data storage devices, and carrier waves(such as data transmission through the Internet).

The configurations and methods of the above-described embodiments areapplied to the video display device without any restriction, and all orsome of the embodiments may be selectively combined to have variousmodifications.

Although embodiments have been described with reference to a number ofillustrative embodiments thereof, it should be understood that numerousother modifications and embodiments can be devised by those skilled inthe art that will fall within the spirit and scope of the principles ofthis disclosure. More particularly, various variations and modificationsare possible in the component parts and/or arrangements of the subjectcombination arrangement within the scope of the disclosure, the drawingsand the appended claims. In addition to variations and modifications inthe component parts and/or arrangements, alternative uses will also beapparent to those skilled in the art.

What is claimed is:
 1. A control method of a video display deviceincluding a display unit, the method comprising: receiving abroadcasting signal through a tuner; obtaining an uncompressed audio orvisual (AV) content other than the broadcasting signal; outputting theuncompressed AV content through the display unit; obtaining ratinginformation on the uncompressed AV content on the basis of a section ofthe uncompressed AV content; and outputting an enhanced service on thebasis of the rating information through the display unit,
 2. The controlmethod according to claim 1, wherein outputting an enhanced servicecomprises outputting the enhanced service for uncompressed AV contentthrough comparison of parental locking information and the ratinginformation of the uncompressed AV content.
 3. The control methodaccording to claim 2, further comprising: outputting a user interfacefor receiving a password input for parental locking information,receiving the password input, and receiving a input for setting theparental locking information including at least one of start time andend time of a parental lock and a rating value for watching availableage, when the received password input is accurate.
 4. The control methodaccording to claim 2, wherein the outputting an enhanced service to theuncompressed AV content comprises blocking the uncompressed AV content,when the rating information on the uncompressed AV content does notsatisfy the set parental locking information.
 5. The control methodaccording to claim 1, wherein the obtaining the rating informationcomprises obtaining the rating information from watermark informationembedded in the section.
 6. The control method according to claim 1,wherein the obtaining the rating information comprises, extractingfeature information of the section and obtaining the rating informationthrough a reply to a query including the feature information.
 7. Thecontrol method according to claim 1, wherein the uncompressed AV contentcomprises a first advertisement, wherein the enhanced service comprisesa second advertisement, and wherein the outputting of the enhancedservice comprises outputting the second advertisement instead of thefirst advertisement.
 8. A video display device comprising: a displayunit; a tuner for receiving a broadcasting signal; a receiving unit forobtaining an uncompressed AV content, other than the broadcastingsignal; an enhanced service management unit for obtaining ratinginformation on the uncompressed AV content on the basis of a section ofthe uncompressed AV content and obtaining an enhanced service on thebasis of the rating information; and a play controlling unit foroutputting the uncompressed AV content and the enhanced service on thedisplay unit.
 9. The video display device according to claim 8, whereinthe enhanced service management unit outputs the enhanced service forthe uncompressed AV content through comparison of parental lockinginformation and the rating information of the uncompressed AV content.(00552,

63-67)
 10. The video display device according to 9, wherein the playcontrolling unit outputs a user interface for receiving a password inputto set parental locking information, and wherein the enhanced servicemanagement unit receives the password input, and receives a input forsetting the parental locking information including start time and endtime of a parental lock and a rating value for watching available age,when the received password input is accurate.
 11. The video displaydevice according to claim 9, wherein the enhanced service managementunit blocks the uncompressed AV content, when the rating information ofthe uncompressed AV content does not satisfy the set parental lockinginformation.
 12. The video display device according to claim 8, whereinthe enhanced service management unit obtains the rating information fromwatermark information embedded in the section.
 13. The video displaydevice according to claim 8, wherein the enhanced service managementunit extracts feature information of the section and obtains the ratinginformation through a reply to a query including the featureinformation.
 14. The video display device according to claim 8, whereinthe uncompressed AV content comprises a first advertisement, wherein theenhanced service comprises a second advertisement, and wherein the playcontrolling unit outputs the second advertisement instead of the firstadvertisement.